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Executive summary 


About the study 


The International Computer and Information Literacy Study 2018 (ICILS 2018) studied the 
extent to which young people are able to use information and communication technology (ICT) 
productively in school, home, society, and their future workplaces. ICILS 2018 builds on methods 
and findings from the first cycle of ICILS, conducted in 2013 (ICILS 2013). 


ICILS 2013 focused on student’s computer and information literacy (CIL) which was defined 
as “an individual’s ability to use computers to investigate, create, and communicate in order to 
participate effectively at home, at school, in the workplace, and in society” (Fraillon et al. 2013, 
p. 17). Put simply, CIL refers to a student’s ability to use computer technologies to collect and 
manage information, and to produce and exchange information. The structure of the CIL construct 
references four strands that frame the skills and knowledge addressed by the CIL assessment: 
understanding computer use, gathering information, producing information, and digital communication. 


ICILS 2018 continued to investigate CIL and added an investigation of students’ computational 
thinking (CT) as an option for participating countries.” CT is the type of thinking used when 
programming on a computer or digital device. In ICILS 2018, CT is defined as “an individual’s 
ability to recognize aspects of real-world problems which are appropriate for computational 
formulation and to evaluate and develop algorithmic solutions to those problems so that the 
solutions could be operationalized with a computer” (Fraillon et al. 2019, p. 27). CT comprises 
two strands: conceptualizing problems (through algorithmic or systems thinking) and operationalizing 
solutions (creating, implementing, and evaluating computer-based solutions to problems). 


ICILS 2018 used a customized assessment software platform that delivered the assessment 
content and a questionnaire about ICT use to students offline. In the majority of schools the 
assessment was delivered from a USB drive. Although the software could have been delivered 
viathe internet, USB delivery ensured a uniform assessment environment for students regardless 
of the quality of internet connections in participating schools. Data were either uploaded to a 
server or delivered to the ICILS research center in that country. 


The ICILS 2018 instrument used purpose-built applications that followed standard interface 
conventions. Students completed a range of tasks, including skills-based tasks using software 
tools (such as text editors or presentation applications) and web content. The purpose-built 
applications were designed to be consistent with the applications that could reasonably be 
expected to be within the realm of students’ typical experience of computer use. 


ICILS 2018 was based around research questions that focused on the following for CIL (in all 
countries) and CT (in countries where CT was also assessed): 


e Variations in CIL and CT within and across countries; 


e Aspects of schools and education systems that are related to student achievement in CIL and 
CT; 


e Relationships of CILand CT with students’ levels of access to, familiarity with, and self-reported 
proficiency in using computers; 


e Aspects of students’ personal and social backgrounds (such as gender and socioeconomic 
background) that are related to students’ CIL and CT; and 


e The relationship between CIL and CT. 


In this report, education systems are usually referred to as “countries.” This is for ease of reading, but it should be noted 
that there are systems that are not countries but are units with a degree of educational autonomy that have participated 


following the same standards for sampling and testing. xvii 


xvili PREPARING 


Four countries participated in both ICILS 2013 and ICILS 2018. It is 
CIL between 2013 and 2018 in the three of those countries th 
requirements for both cycles. 


Data collection 
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possible to compare student 
at met the ICILS technical 


ICILS 2018 gathered data from 46,561 grade 8 (or equivalent) students in more than 2226 
schools from 12 countries and two benchmarking participants. These student data were 


augmented by data from 26,530 teachers in those schools and b 


Hemisphere. 


yy contextual data collected 


from school ICT coordinators, principals, and national research centers. Eight of the countries 
and one benchmarking participant participated in the optional CT assessment. 


The ICILS 2018 main survey data collection took place in the first half of 2018 for participants 
in the Northern Hemisphere and the second half of 2018 for participants in the Southern 


ICILS collected data using six instruments (seven in countries that participated in the CT 
assessment). Students completed the test of CIL, a questionnaire, and (where applicable) the 


test of CT. Separate questionnaires were completed by teachers, sch 
principals, and staff in national research centers. 


Assessing CIL and CT 


ool ICT coordinators, school 


ICILS 2018 measured students’ ability to use computers to collect and manage information, and 
to produce and exchange information (CIL), as well as formulate solutions to problems so that 
those solutions could be operationalized with a computer (CT). In ICILS 2018 the two domains 


are regarded as complementary aspects of a broader notion of dig 
assessed these domains through computer-based assessments ba 
and problems. It investigated variations in CIL and CT across an 
relationships between each construct and student attributes (bac 
developed attributes), including their use and experience of com 
contexts in which CIL and CT are developed. ICILS 2018 also in 
between CIL and CT. 


ital competence. ICILS 2018 


sed on real-world scenarios 
d within countries, and the 
keround characteristics and 
puter technologies and the 
vestigated the associations 


The ICILS 2018 test instrument tasks were embedded within modules. In total, there were five 


30-minute CIL modules and two 25-minute CT modules. Each student completed two of the 


five available CIL modules and (where applicable) the two CT mod 


allocated to students in a balanced randomized design. The order in which CT modules were 
presented was randomly allocated to students. In countries participating in the CT option, 
the CIL assessment and the 


students completed the CT modules after having finished both 
student questionnaire. 


ClL modules consisted of a sequence of tasks contextualized by a real-world theme. Each module 
was a series of five to eight smaller tasks, each of which typically took students less than one 
minute to complete, and a single large task which typically took 15 to 20 minutes to complete 
and involved the development of an information product. The large tasks were specified for 
students in terms of the software tool and format to be used, the communicative purpose, and 
the target audience of the information product. Three of the CIL modules had been used in 
ICILS 2013 and kept secure. Two new modules were developed for the ICILS 2018 CIL test 


ules. The CIL modules were 


instrument. Data collected from all five CIL modules were used as the basis for reporting ICILS 


2018 ClL results on the ICILS CIL achievement scale. 
Two 25-minute CT modules were developed for the ICILS 2018 


CT assessment. Each had a 


unifying theme, and a sequence of tasks that related to the theme (but not a large task). The tasks 


inthe CT module focusing on conceptualizing problems related to p 
to operate adriverless bus. This included the visual representation of 


lanning aspects of a program 
real-world situations in ways 
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that could support the development of computer programs to execute automated solutions (e.g., 
path diagrams, flow charts, and decision trees). Further tasks related to the use of simulations to 
collect data and draw conclusions about real-world situations that could inform the development 
of acomputer program. In the CT module focusing on operationalizing solutions, students worked 
within a simple visual coding environment to create, test, and debug code that controlled the 
actions of a drone used in a farming context. In this module, the tasks were incrementally more 
complex as the students progressed through the module. The complexity of the tasks related 
to the variety of available code functions and the sequence of actions required by the drone 
for completion of the task. 


Collecting data on students’ personal and educational contexts for 
developing CIL and CT 


As had been the case for ICILS 2013, a secondary purpose of ICILS 2018 was to investigate 
the use of computers and other digital devices by students and teachers, as well as students’ 
and teachers’ attitudes toward the use of digital technologies. Some of these aspects of ICT 
use are potentially related to student outcomes, while others may not be directly associated 
with them but may inform understanding of the broader context in which digital technologies 
are used, including contexts both inside and outside of school in which CIL and CT are learned. 


CILS 2018 investigated student engagement with ICT through a 30-minute student questionnaire 
completed on computer by students following the CIL assessment. It included questions relating 
to students’ experience and use of ICT to complete a range of different tasks in school and out 
of school, their attitudes towards the use of computers and ICT, and background characteristics. 


CILS also included a teacher questionnaire that was completed by a random sample of 15 
teachers of grade 8 students in each school. The questionnaire asked teachers about their 
familiarity with ICT, their use of ICT in educational activities in teaching focused on a randomly- 


selected reference class, teachers’ perceptions of ICT in schools, learning to use ICT in teaching, 
and their background. 


A 15-minute questionnaire completed by ICT coordinators asked about ICT resources in the 
school, |CT use in the school, ICT technical support, and provisions for professional development 
in ICT. A 15-minute questionnaire completed by school principals provided information 
about school characteristics and policies, procedures, and priorities for ICT. National research 
coordinators collated information provided by experts about contextual factors concerned with 
the structure of the education system and other aspects of education policy concerning ICT- 
related learning. 


Findings 


National contexts for CIL and CT education 


The plans and policies of countries for improving student learning and ICT resources, methods 
to support student learning, and the priorities for the use of ICT were similar. Although many 
countries recognized aspects of CIL in curricula (in nine of the 14 participants as part of a 
designated subject) aspects of CT were less commonly included. There were large differences 
across countries in terms of the availability of ICT devices for students. 


Student CIL 


CIL achievement was described across four levels of increasing sophistication. However, 18 
percent of students on average across all countries were working below the lowest proficiency 
level, which meant that they did not have a functional working knowledge of computers as tools 
and are unlikely to be able to create digital information products unless they have support and 
guidance. 
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Twenty-five percent of students were working at Level 1, which meant they demonstrated a 
functional working knowledge of computers as tools. 


ng at Level 2, and could use computers, under direct 
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Aspects of student background were also associated with CIL. Female students demonstrated 
higher CIL achievement than male students. Socioeconomic status was significantly positively 
associated with student CIL achievement. Students from non-immigrant families had higher CIL 
scores than students from immigrant families. Students’ CIL varied more within countries than 
across countries. 
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Students’ engagement with ICT 


Slightly fewer than half of the students in grade 8 had been using computers for five or more 
years. We called these students experienced computer users. Seven out of 10 students used 
ICT on a daily basis outside school for general purposes (including school-related and other 
purposes) but only one student in five reported using ICT daily for school-related purposes. 


Student use of general applications (such as word processing, presentation, or internet search 
tools) was more frequent among experienced computer users, those currently studying a computer 
subject at school, and those with higher levels of CIL. 


Student use of ICT for leisure activities was common. Most used ICT at least once each week 
to download music or watch videos. Approximately two thirds of students used ICT to access 
information of interest from the internet at least once each week. 


The most frequent school-related use of ICT was using the internet to do research. Approximately 
three students in five did this at least once per week. About a quarter of the students used ICT 
on a weekly basis to work with other students, organize their time and work, or prepare reports 
and essays. The most commonly-used ICT tools were computer-based information resources, 
word processing software, and presentation software. 


n 
U 


Four out of five students were confident about their ability to use ICT to search for information, 
insert an image into a document, and write or edit text for a school assignment. Most students 
acknowledged positive outcomes of ICT for society, but around half of the students also agreed 
that ICT had some negative consequences for society. 


Teaching with and about ICT 


On average across countries, most teachers tended to have experience and familiarity with ICT. 

ore than two thirds of teachers on average had at least five years of experience with the use of 
CT during lessons or with their preparation. However, fewer than half of the teachers reported 
frequent use of ICT when teaching (though there was considerable variation in the frequency 
of pedagogical ICT use across participating countries). 


ost teachers tended to recognize the value of ICT use for teaching and were confident about 
using this technology. Majorities of teachers expressed confidence when undertaking a large 
number of |CT-related tasks, however teachers were less confident regarding their use of online 
discussions, online collaboration, and the use of learning management systems. Teachers who 
used ICT frequently for teaching were more confident about using ICT and older teachers felt 
ess confident about using ICT. 


ost teachers endorsed the advantages of ICT use for student learning but there were also 
considerable proportions of teachers mindful of negative effects. Teachers who reported daily 
CT use for teaching tended to be less concerned about its potential negative effects. 


There were considerable differences across countries in the availability of ICT at schools, the 
extent of teacher collaboration, and conditions for professional learning. Teachers who used ICT 
more frequently in class were also more positive about teacher collaboration and the quality of 
ICT resources at school. 


Most teachers emphasized developing skills related to CIL and CT. However, there was 
considerable variation in teacher emphasis across different subject areas. Teachers’ confidence 
in using ICT, positive perceptions of pedagogical ICT use, and positive perceptions of teacher 
collaboration were positively associated with teacher emphasis on student learning of ClL and CT. 


Teachers’ use of ICT in classrooms was still limited and depended on the complexity of tasks and 
applications. Teachers reported higher levels of use of general ICT tools than digital learning tools. 
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Explaining variations in CIL and CT 


ICILS 2018 reports the outcomes of multivariate and multilevel models used to explain variations 
in CIL and CT within countries. Student-level and school-level influences were incorporated in 
separate models explaining variation in CIL and CT outcomes. 


There were considerable differences in the variance for both CIL and CT as well as the proportion 
of variance found between schools across participating countries. Most of the variance in CIL and 
CT scores was explained at the school level, with less variance being accounted for within schools. 


Personal and social background had consistent net effects on CIL and CT. Female gender tended 
to be positively related to CIL scores, but negatively related to CT scores. Home use of test 
language was a positive predictor of CIL and CT in some countries, in particular in those with 
larger proportions of students speaking another language at home. Expected university education 
and socioeconomic background were consistent positive predictors of both CIL and CT ability. 


Several student-level variables related to experience and use of ICT had consistent net effects on 
CIL and CT. Students’ daily use of ICT and experience with computers were consistent positive 
predictors of both CIL and CT. The availability of computers at home was a positive predictor in 
most countries, but after controlling for personal and social background the effect was weaker. 
Student reports on having learned about ClL-related tasks at school and student reports on the 
use of general ICT applications in class tended to be positive predictors of student CIL in severa 
countries. However, students who reported having been taught CT-related tasks had lower CT 
scores than other students. 


There were mostly inconsistent net effects on CIL and CT by school-level |CT-related variables. 
Schools’ socioeconomic context was a consistent positive predictor in almost all countries. In 
the few countries where school-level factors had significant associations, these were often no 
longer significant after controlling for social context variables. 
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CHAPTER 1: 


Introduction to the IEA International 
Computer and Information Literacy Study 
2018 


Background 


The International Association for the Evaluation of Educational Achievement (IEA) has been 
researching the impact of information and communications technologies (ICT) on educational 
processes, and factors supporting or impeding the pedagogical use of ICT, since the late-1980s. 
More recently it has turned its attention to investigating |CT-related educational outcomes. IEA’s 
International Computer and Information Literacy Study (ICILS) was developed in response to 
the increasing use of ICT in modern society and the need for people to have the capabilities 
necessary to participate effectively in a digital world. 


The first cycle of ICILS in 2013 (ICILS 2013) assessed students’ computer and information literacy 
(CIL) which brings together technical competence with computer applications and the capacity 
to manage information (Fraillon et al. 2014). This places an emphasis on the use of computers 
as information seeking, management, and communication tools which are key to participation 
in the digital age (see, for example, Chalkiadaki 2018; van Laar et al. 2017). 


This second cycle of ICILS in 2018 (ICILS 2018) continued to investigate students’ CIL but also 
investigated students’ computational thinking (C his dimension involves conceptualizing 
problems (through algorithmic or systems thinking) and operationalizing solutions (creating, 
implementing, and evaluating computer-based responses to problems). The inclusion of CT as 
an option in ICILS 2018 reflects recent interest by educators, researchers, and policymakers in 
the value of CT in schooling. ICILS 2018 studied how these components of digital competence 
related to each other and to the school and out-of-school contexts that support learning with 
and about computer technology. 


This report presents the outcomes at the international level of analyses of data collected in the 
ICILS main survey in 2018. The report aims to provide an international perspective on the ICILS 
data relating to CIL collected across countries. Twelve countries participated in ICILS 2018: Chile, 
Denmark, Finland, France, Germany, Italy, Kazakhstan, the Republic of Korea (hereafter referred 
to as Korea, for ease of reading), Luxembourg, Portugal, the United States, and Uruguay. The city 
of Moscow (Russian Federation) and the state of North Rhine-Westphalia (Germany) participated 
as benchmarking participants. Eight countries completed the CT assessment: Denmark, Finland, 
France, Germany, Korea, Luxembourg, Portugal, and the United States. The CT option was also 
completed by the state of North Rhine-Westphalia (Germany) as a benchmarking participant. This 
report responds to the ICILS research questions and provides observations and interpretations 
that may stimulate further investigation within and across countries. 


Purposes of ICILS 2018 


The primary purpose of ICILS 2018 was to assess the capacities of students to use ICT 
productively for a range of purposes, including those that go beyond a basic use of computers 
such as producing information products and managing digital information. Authentic computer- 
based assessments administered to students in their eighth year of schooling generated data 
that provided measures of two domains of digital competence: CIL and CT. 


41 Benchmarking participants are education systems within countries. 


© IEA International Association for the Evaluation of Educational Achievement 2020 
J. Fraillon et al., Preparing for Life in a Digital World, 
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National Center for Education Statistics 2016). ICILS 2013 invoked the term CIL to emphasize that 
having the capacity to use the internet to search for and evaluate information was an important 
part of the broad capability to use modern technology (Fraillon et al. 2013). 


Computational thinking 


CT is the type of thinking used when programming on a computer or developing an application 
for another type of digital device. Fraillon et al. (2019, p. 27) defined CT as “an individual’s ability 
torecognize aspects of real-world problems which are appropriate for computational formulation 
and to evaluate and develop algorithmic solutions to those problems so that the solutions could 


be 


CT focuses on learning foundational pr 


operationalized with a computer.’ 


inciples of computing. Wing (2006) regarded CT as a 


concept that embraces problem solving and system design, based on principles central to computer 


Sci 
(G 
to 


2012, p. 29 


ence. CT has also been described as the ways of thinking when programming a computer 
rover and Pea 2013) and can be seen as “applying tools and techniques from computer science 
understand and reason about both natural and artificial systems and processes” (Royal Society 
. Shute et al. (2017) argued that CT is required to solve problems algorithmically 
ith or without the assistance of computers) by applying solutions that are reusable in different 
ntexts. They suggested that CT involves six elements: decomposition, abstraction, algorithm 
sign, debugging, iteration, and generalization. CT does not necessarily involve developing or 
p 


ementing a formal computer code (Barr et al. 201.1). However, assessments of CT are typically 
t in computer environments because those facilitate the capturing of the data that reflect the 


steps in problem solving. These steps usually involve developing or assembling instructions (often 


Th 
n 


A 
O 
in 
an 


including 
Yadav et al. 
f “abstraction, algorithms and automation.’ 


O 
O 
programming and problem solving are important for educational deve 
Cc 
r 


blocks of code) that are necessary to accomplish a task (Brennan and Resnick 2012). 
2018, pp. 91-92) articulated the nature of CT as being focused on the processes 


e early stages of the introduction of computers in schools and classrooms included a focus 
programming (Lockheed and Mandinach 1986). It can be argued that the links between 
opment (Papert 1980). 
ey element of this in the 1980s was the educational programming language Logo, in which 
mmands resulted in movement of a cursor or robot (termed a “turtle”) on a screen, producing 
e graphics. Recently there has been a resurgence of interest from researchers, educators, 
d policymakers in the importance of CT in education (Voogt et al. 2015). Visual programming 


languages (where programs are created by manipulating program elements, or blocks, graphically) 
for children have emerged in addition to text-based programming languages (e.g., Scratch and 
Python) (Ortiz-Colon and Marato Romo 2016). These languages focus on the algorithmic logic 


un 


derpinning coding across tasks and are considered to be accessible to novice users. 


Research questions 


IC 
of 
us 
re 


LS aimed to investigate the extent of CIL and CT among grade 8 students, and the associations 
these learning outcomes with student background, developed attributes, experience with 
ing computer technologies, and learning about computer technologies. It also investigated 
ations between CIL and CT. 


Computer and information literacy 


Th 
qu 


e research questions concerned with CIL remain similar to those used in ICILS 2013. The 
estions are framed around variations in CIL, the relationship of CIL to the characteristics of 


students, and the contexts in which CIL is developed. These have been articulated more fully 


in 


the ICILS 2018 assessment framework (Fraillon et al. 2019). In summary, the four research 


qu 


estions were the following: 
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RQCIL1 What variations exist across and within countries in students’ CIL? 


RQCIL2 What aspects of schools and countries are related to students’ CIL? 


RQCIL3 What are the relationships between students’ levels of access to, familiarity with, 
and self-reported proficiency in using computers and their CIL? 


RQCIL4 What aspects of students’ personal and social backgrounds (such as gender and 


socioeconomic bac 


kground) are related to students’ CIL? 


Four countries participated in each of ICILS 2013 and ICILS 2018. It is possible to compare 


student CIL between 2013 and 
requirements for both cycles. 


Computational thinking 


2018 in the three of those counties that met the ICILS technical 


The research questions relating to CT closely reflected those proposed for CIL, but excluded 
reference to changes from ICILS 2013 and included reference to the relationship between CT 


and CIL. Analyses were limited 


to those countries participating in the optional assessment of 


students’ CT achievement. The five CT related research questions were the following: 


socioeconomic bac 


RQCT1 What variations exist across and within countries in students’ CT? 
RQCT2 What aspects of schools and education systems are related to students’ CT? 


RQCT3 What are the relationships between students’ levels of access to, familiarity with, 
and self-reported proficiency in using computers and their CT? 


RQCT4 What aspects of students’ personal and social backgrounds (such as gender and 


kground) are related to students’ CT? 


RQCT5 What is the association between students’ CIL and CT? 


The ICILS assessment framework 


The ICILS 2018 assessment framework provides a conceptual underpinning for the international 
instrumentation for ICILS (Fraillon et al. 2019). The assessment framework consists of three parts: 


e The CIL framework outlines the outcome measures addressed through the CIL test. 


e The CT framework outlines the outcome measures addressed through the CT test. 


e Thecontextual framework maps the context factors expected to influence and explain variation 


nClL and CT. 
The CIL framework 


The structure of the CIL constr 


uct references four strands that frame the skills and knowledge 


addressed by the CIL assessment: understanding computer use, gathering information, producing 


information, and digital commu 
intended to represent empirica 
of two aspects (Figure 1.1). 


e Understanding computer use 
underpin the operational use 


nication. These strands define the CIL construct but are not 
ly distinct components. Each strand is further defined in terms 


refers to the fundamental technical knowledge and skills that 
of computers as tools for working with information. This includes 


a person’s knowledge and understanding of the generic characteristics and functions of 
computers. Understanding computer use comprises two aspects: foundations of computer 
use and computer use conventions. 


e Gathering information embraces the receptive and organizational elements of information 
processing and management. This subsumes two aspects: accessing and evaluating 
information, and managing information. 
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e — Producing information focuses on using computers as tools for thinking and creating. It 


involves two aspects: transforming information and creating information. 


e — Digital communication focuses on information sharing in social networking (and broader web- 
based information sharing spaces) together with the social, legal, and ethical responsibilities 


associated with information sharing and using information safely and securely. 


Figure 1.1: ICILS 2018 CIL framework 


Strand 1 
Understanding 
computer use 


Aspect 1.1 
Foundations of 
computer use 


Aspect 1.2 
Computer use 
conventions 


Source: Fraillon et al. (2019). 


The CT framework 


Strand 2 
Gathering information 
Aspect 2.1 


Accessing and 
evaluating information 


Aspect 2.2 
Managing information 


Strand 3 
Producing information 
Aspect 3.1 


Transforming 
information 


Aspect 3.2 
Creating information 


Strand 4 

Digital communication 
Aspect 4.1 

Sharing information 


Aspect 4.2 
Using information 
responsibly and safely 


The CT construct comprises two strands: conceptualizing problems and operationalizing solutions 
(Figure 1.2). One strand contains three aspects and the other comprises two aspects. The 
aspects encompass the knowledge, skills, and understandings held in common across the range 


of definitions of CT. 


¢ Conceptualizing problems acknowledges that before solutions can be developed, problems 
must first be understood and framed in a way that allows algorithmic or systems thinking to 


assist in the process of developing solutions. This strand comprises three aspects: knowing 
about and understanding digital systems, formulating and analyzing problems, and collecting 


and representing relevant data. 


¢ Operationalizing solutions comprises the processes associated with creating, implementing, and 
evaluating computer-based system responses to real-world problems. It includes the iterative 
processes of planning for, implementing, testing, and evaluating algorithmic solutions (as the 
potential bases for programming) to real-world problems. The strand includes an understanding 


of the needs of users and their likely interaction with the system under development. This 
strand comprises two aspects: planning and evaluating solutions, and developing algorithms, 


programs, and interfaces. 
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Figure 1.2: ICILS 2018 CT framework 


Strand 1 Strand 2 

Conceptualizing problems Operationalizing solutions 

Aspect 1.1 Aspect 2.1 

Knowing about and understanding digital Planning and evaluating solutions 
systems Aspect 2.2 

Aspect 1.2 Developing algorithms, programs, and 
Formulating and analyzing problems interfaces 

Aspect 1.3 


Collecting and representing relevant data 


Source: Fraillon et al. (2019). 


The contextual framework 


ICILS 2018 collected contextual information so as to provide bases for understanding variations 
in ClL and CT (from here on we use CIL/CT to refer to these outcomes when common contextual 
information is being considered) as well as to collect data about the pedagogical use of ICT 
at schools. We classified the contextual factors in a framework that was consistent with the 
multilevel structure inherent in student CIL/CT learning and also considered these factors as 
antecedents or processes (Figure 1.3). 


Conceptual frameworks for analyzing educational outcomes are frequently based on a multilevel 
structure that is inherent in student learning (Goldstein 2010; Raudenbush and Bryk 2002; 
Schulz 2018). Learning is set in the overlapping contexts of in-school and out-of-school learning. 
Out-of-school activities and experiences are important for CIL/CT learning (ACARA [Australian 
Curriculum, Assessment and Reporting Authority] 2015). The contextual framework of ICILS 
therefore embraces out-of-school learning through experience and frequency of ICT use by 
students outside of school (Fraillon et al. 2019). 


The temporal status of contextual factors within the learning process is also important. Factors 
may be considered either as antecedents or processes. Antecedents are exogenous factors that 
condition the ways in which CIL/CT learning takes place. They are contextual factors that are 
not directly influenced by learning-process variables or outcomes. It is important to recognize 
that antecedent variables are level-specific and may be influenced by antecedents and processes 
found at higher levels, for example, the extent to which schools’ ICT resources are likely to be 
influenced by ICT education policies at the level of the education system. Processes are those 
factors that directly influence CIL/CT learning. They are constrained by antecedent factors 
and factors found at higher levels. These antecedent factors could include variables such as 
opportunities for CIL/CT learning during class, teacher attitudes toward using ICT for study 
tasks, and students’ use of computers at home. 
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Figure 1.3: Contexts for ICILS 2018 CIL/CT outcomes 


Antecedents Processes Outcomes 


School/classroom 
Characteristics 
Stated ICT curriculum 
ICT resources 


School/classroom 
ICT use for teaching/ 
learning 

CIL/ICT instruction 


Student 
Learning process 


Student 
Characteristics 


Notes: The double arrow between process-related factors and outcomes emphasizes the possibility of a reciprocal 
association between learning processes and learning outcomes. The single-headed arrow between antecedents and 
processes indicates the assumption within the ICILS contextual framework of a unidirectional influence between these 
two types of contextual factors. 


Source: Fraillon et al. (2019). 


Reference to this general conceptual framework enabled us to locate potential contextual factors 
on a two-by-four grid where antecedents and processes constitute the columns and the four 
levels the rows (Table 1.1 provides examples of the contextual variables collected by the ICILS 
instruments). 


Table 1.1: Mapping of variables to the contextual framework related to CIL and CT outcomes (examples) 


Level of... Analysis level Antecedents Processes 
Wider community Not used in NCS & other sources: NCS & other sources: 
within-country Structure of education Role of ICT in curriculum 
multilevel analyses Accessibility of ICT 
School/classroom School level PrQ, ICQ, & TQ: PrQ, ICQ, TQ, & StQ: 
School characteristics ICT use in teaching 
ICT resources and learning 
CIL/CT instruction 
Student Student level StQ: StQ: 
Gender ICT activities 
Age Use of ICT 
ICT experience CIL/CT 
Home environment StQ: StQ: 
Parent socioeconomic Learning about ICT 
status at home 
Home ICT resources 


Note: NCS = national contexts survey; PrQ = principal questionnaire; |CQ = ICT coordinator questionnaire; TQ 
= teacher questionnaire; StQ = student questionnaire. 
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The student questionnaire collected data on contextual factors pertaining to the level of the 
individual student and their home context. The teacher, school principal, and ICT coordinator 
questionnaires were designed to gather information about contextual factors associated with 
the school/classroom level. In addition, and separate from the multilevel analyses, the national 
contexts survey (NCS) and other available sources (e.g., published statistics), provided national 
contextual data that facilitated interpretation. 


ICILS instruments 


In total, ICILS collected data using six instruments (seven in countries that participated in the 
CT assessment). Students completed the test of CIL, a questionnaire, and (where applicable) 
the test of CT. Separate questionnaires were completed by teachers, school ICT coordinators, 
school principals, and staff in national research centers. 


The ICILS 2018 assessments of students’ CIL and CT were designed to provide students with 
an authentic computer-based assessment experience in a uniform way. ICILS 2018 used a 
customized assessment platform that delivered the assessment content to students offline. In 
the majority of schools the assessments were delivered from a USB drive. A few schools installed 
the materials on a computer that functioned as a server for a network of school computers. 
The instrument used purpose-built applications that followed standard interface conventions. 
Students completed a range of tasks including skills-based tasks using productivity software 
tools (such as text editors or presentation applications) and web-content. The purpose-built 
applications were designed to be consistent with the applications that could reasonably be 
expected to be within the realm of students’ typical experience of computer use. 


CIL test design 
The ClL assessment tasks were embedded within modules. In total, there were five 30-minute CIL 
modules. Each student completed two of the five CIL modules. The CIL modules were allocated 
to students in a balanced randomized design. 


Each CIL module comprised a sequence of tasks contextualized by a real-world theme and driven 
by a plausible narrative. Each module included a series of five to eight smaller tasks, each of 
which typically took students less than one minute to complete, and each of which contributed 
to the development of contextual knowledge that underpinned work on a single large task. 
The large tasks typically took 15 to 20 minutes to complete and involved the development 
of an information product (such as a presentation, poster, website, or social media post) that 
made use of information and resources managed by students in the lead-up tasks. The large 
tasks were specified for students in terms of the software tools and format to be used (and 
consequently the format of the product), the communicative purpose, and the target audience 
of the information product. 


Three of the CIL modules had been developed and used in ICILS 2013 and kept secure. Two 
new modules were developed for the ICILS 2018 CIL test instrument to address contemporary 
thematic content and software environments. Data collected from all five CIL modules in ICILS 
2018 were used as the basis for reporting ICILS 2018 CIL results on the ICILS CIL achievement 
scale established in 2013. The rotated module design enabled the instrument to contain, and 
consequently report on achievement against, a larger amount of content (covering the breadth 
of the CIL framework and a range of difficulties) than any single student could reasonably 
complete in 60 minutes. 


INTRODUCTION TO IEA ICILS 2018 
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National coordinator questionnaires 


ICILS 2018 national research coordinators provided information, 


based on the input of national 


experts, in response to an online NCS. Data from the NCS were used for comparing profiles of 
CIL and CT education in participating countries. The NCS provided data on contextual factors 


concerned with structure of the education system and systema 
practice in the use of ICT in school education. The data provid 
the influence of system-level contexts on differences in CIL/CT 
differences among countries in the patterns of relationships amo 
CIL/CT learning. 


tic descriptions of policy and 
ed the bases for analyses of 
earning, and for interpreting 
ng factors that are related to 


Participating countries, population, sample design, and achieved 


samples 


Countries or education systems 


Twelve countries and two benchmarking participants participated in ICILS 2018. 

Countries participating in ICILS 2018 

e Chile (CIL) e Kazakhstan (CIL) 

e Denmark (ClL and CT) e Korea (ClL and CT) 

e Finland (ClL and CT) e Luxembourg (CIL and CT) 

e France (CIL and CT) e Portugal (CIL and CT) 

e Germany (ClL and CT) e The United States (CIL and CT) 

e Italy (CIL) e Uruguay (CIL) 

Benchmarking participants 

e oscow (Russian Federation) (CIL) = « North Rhine-Westphalia (Germany) (CIL and CT) 


Benchmarking participants are education systems within countri 


es. In ICILS 2018, the city of 


Moscow in the Russian Federation took part as a benchmarking participant even though the 
country did not participate. In contrast, the German state of North Rhine-Westphalia took 


part as a benchmarking participant in addition to the participatio 


n of the country of Germany. 


Additional schools were sampled in North Rhine-Westphalia to enable accurate reporting of 
data representing that entity. Data collected from North Rhine-Westphalia also contributed to 


the data reported for Germany as a whole. 


the benchmarking participant has satishled the technical require 


Data collected from benchmarking participants were not included in the establishment 
of reporting scales, nor were they included in the computation of international averages. 
However, data from a benchmarking participant may be compared to the internationa 


data if 
ments of the study. Both the 


city of Moscow (Russian Federation) and North Rhine-Westphalia (Germany) met the technical 


requirements of ICILS 2018. 


Administration periods 


The ICILS 2018 main survey data collection took place in the first half of 2018 for participants 
in the Northern Hemisphere? and in the second half of 2018 for participants in the Southern 


Hemisphere. In Italy the survey data collection took place in the 


second half of 2018 (i.e., the 


beginning of the school year) even though it was a Northern Hemisphere participant. 


2 The Northern Hemisphere participants were: Denmark, Finland, France, Germany, Italy, Kazakhstan, Korea, Luxembourg, 


Portugal, the United States, Moscow (Russian Federation), and North Rhine-Wes 
3 The Southern Hemisphere participants were Chile and Uruguay. 


tphalia (Germany). 
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Population definitions 


The ICILS student population was defined as students in grade 8 (typically around 14 years of 
age in most countries), provided that the average age of students in this grade was at least 13.5 
years at the time of the assessment. 


The population for the ICILS teacher survey was defined as consisting of all teachers teaching 
regular school subjects to the students in the target grade at each sampled school. It included 
only those teachers who were teaching the target grade during the testing period and who had 
been employed at the school since the beginning of the school year. ICILS also administered 
separate questionnaires to principals and nominated ICT coordinators in each school. 


Sample design 
Schools 


The samples were designed as two-stage cluster samples. During the first stage of sampling, 
schools with enrolled students at the target grade were randomly selected with a probability 
proportional to size as measured by the number of students enrolled in a school. The numbers 
required in the sample to achieve the necessary precision were estimated on the basis of national 
characteristics. However, as a guide, each country was instructed to plan for a minimum sample 
size of 150 schools except in very small education systems where all schools were included in 
the survey. The schools sampled at the first stage were then used to select both students and 
teachers. The numbers of schools in the achieved samples in each country or benchmarking 
participant ranged between 35 (in a very small system, with next smallest sample being 110) 
and 261. 


Students 


Within each participating school, 20 students were randomly sampled from all students enrolled in 
the target grade. In schools with fewer than 20 students, all students were invited to participate. 


Teachers 


Fifteen teachers were selected at random from all teachers teaching the target grade at each 
sampled school. In schools with 20 or fewer such teachers, all teachers were invited to participate. 
Because of the intention that teacher information should not be linked to individual students, all 
teachers of the target grade were eligible to be sampled regardless of the subjects they taught. 


Participation requirements and reporting 


The participation rates required for each country were 85 percent of the selected schools and 
85 percent of the selected students within the participating schools, or a weighted overall 
participation rate of 75 percent. The same criteria were applied to the teacher sample, but the 
coverage was judged independently of those for the student sample. 


nthe tables in this report, we use annotations to identify those countries that met these response 
rates only after using replacement schools. Countries that did not meet the response rates, 
even after replacement, are reported separately below the main section of each table. Results 
from education systems that took part as benchmarking participants also appear as a further 
separate section in the tables of this report. (Appendix A documents sampling information and 
participation rates for each country.) 


Achieved samples 

CILS 2018 gathered data from 46,561 grade 8 (or equivalent) students in 2226 schools from 12 
countries and two benchmarking participants. These student data were augmented by data from 
26,530 teachers in those schools and by contextual data collected from school ICT coordinators, 
principals, and national research centers. 
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The average student participation rate (after replacement procedures) achieved was 87 percent. 
Eleven of the 12 participating countries and both benchmarking participants satisfied or nearly 
satisfied the required participation rate for students and are reported in the main section of 
the reporting tables. Italy assessed its grade 8 students at the beginning of the school year and 
therefore the average age of sampled students is lower than 13.5 years at the time of assessment 
which is below target group population age defined for the survey. As a consequence, CIL results 
from Italy are reported separately in the tables in Chapter 3 to indicate to readers that results 
from this country are not entirely comparable with other countries. 


The average teacher participation rate (after replacement procedures) was 82 percent. Seven 
of the 12 participating countries and both benchmarking participants satisfied the required 
participation rate for teachers. 


Structure of this report 


The six chapters following this present and discuss analyses of the ICILS data and serve to address 
the ICILS research questions. The last chapter (Chapter 8) concludes the report with reflections 
onthe themes that are evident across ICILS 2018 and on future directions for research. 


Chapter 2 describes the national contexts for CIL and CT education in ICILS 2018 countries. It 
addresses common patterns as well as policies, curriculum, resources, and practices in specific 
countries and groups of countries. 


Chapter 3 reports on CIL proficiency across countries. It describes how the student test was used 
to measure CIL and presents the ICILS scale of CIL proficiency. The chapter then documents 
how student achievement on the CIL scale varied across and within participating countries. It 
examines gender differences in CIL and variations in CIL related to socioeconomic background, 
immigrant status, and home ICT resources. For three countries it was possible to report on 
changes in CIL between 2013 and 2018. 


Chapter 4 reports on CT proficiency across countries. It describes the nature of the CT tests 
and the achievement scale that was derived from the assessment data. It documents how CT 
achievement varied across and within participating countries and the how variations related to 
gender, socioeconomic background, immigrant status, and home ICT resources. It also reports 
on the association between CIL and CT. 


Chapter 5 explores students’ use of, and engagement with, |CT at home and school. Data reported 
on in this chapter were collected using the student questionnaire. Standardized scale indices 
are used to report students’ use of ICT for a range of purposes and |CT-related attitudes. The 
chapter also reports on differences between male and female students as well as differences in 
use across subject areas, and associations of self-perceptions with CIL/CT achievement. 


The focus of Chapter 6 is teaching with and about ICT, specifically on the roles of schools in 
CIL/CT education. This chapter is based on data from the teacher, ICT coordinator, and principa 
questionnaires and describes the variation in approaches to the provision of CIL/CT related 
education in schools. It reviews teacher familiarity with ICT, teacher confidence in using ICT, 
teacher views about using ICT in education, and the ICT tools that they use in teaching activities. 
It reports on the emphasis teachers place on developing CIL/CT, as well as on the pedagogica 
use of ICT, and associations between these emphases and teacher attributes. 


Chapter 7 presents the outcomes of multivariate and multilevel models used to explain variations 
in CIL/CT within countries. The models incorporate student-level and school-level influences 
explaining variation in CIL and CT outcomes. 
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Chapter 8 discusses the themes emerging from the results of ICILS 2018. We reflect on the key 
findings relating to student achievement in CIL and CT, the digital divide and student gender, and 
on the use of ICT in schools and teaching. The chapter includes reflections on implications of the 
results for policy and practice and suggests some directions for future research on CIL and CT. 
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CHAPTER 2: 


The contexts for education on computer 


and information literacy and computational 


thinking 


Chapter highlights 


Characteristics of the educational systems vary considerably across participating ICILS 
2018 countries. 


In 11 of the 14 ICILS 2018 educational systems (the exceptions being Germany, the 
United States, and benchmarking entity, North Rhine-Westphalia, Germany), the national 
educational ministry leads the primary role of defining the goals and direction for the school 
education system. The characteristics of these systems in terms of years of schooling at 
each educational level and school financing type fluctuated across countries. (Table 2.1) 


n almost all countries, schools had at least some autonomy with most aspects of school 
policies, with private schools typically having a greater degree of autonomy. (Table 2.2) 


e A wide range of differences across participating countries exist, both in relation to 


A 


information and communication technology (ICT) infrastructure availability and economic 
characteristics. (Table 2.3) 


though the formulation of plans and policies supporting the use of ICT in education 


differed across countries, there was a high degree of similarity in the content related to 
improving student learning, ICT resources, methods to support student learning, and the 


priorities for the use of ICT. 


While many countries had explicit or implicit recognition of different computer and 
information literacy (CIL) aspects in their national curriculum,’ aspects of computational 
thinking (CT) were less frequently included. (Table 2.4 and Table 2.5) 


ClL was offered to students at lower-secondary level in all 14 participating countries and 
benchmarking participants. In eight out of 14 this was a separate subject, although it was 
not always compulsory. (Table 2.6) 


Countries had very different approaches to the development of teachers’ capacity to use 
CT. In most countries it was either a mandatory component of pre-service education or 
part of some form of professional development for teachers. Rarely was it a requirement 
for registration as a teacher. (Table 2.7) 


n general, countries provided a large degree of support for teacher access to IC T-based 
professional development, mainly by funding teacher participation in programs and/or 
by providing resources for teachers to access. (Table 2.8) 


4 There is no national curriculum for the United States. Data related to the curriculum reported in this chapter are 


based on selected state curricula. 
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Reports from school principals and ICT coordinators provide a contrasting profile of 
differences across participating ICILS countries in terms of school resourcing, policies, and 
priorities. 


Most technology-related resources and software-related resources were reported as 
being available in schools. (Table 2.9 and Table 2.10) 


Considerable variation was evident across countries on whether ICT facilities were 
available for the teaching and learning of target grade students. (Table 2.11) 


Large differences were evident across countries in terms of the availability of ICT devices 
per student. The overall influence of school location on this ratio was minimal. (Table 
DAD) 


School computers were typically available in computer laboratories. When students were 
able to bring portable computers to class, these were most commonly provided by the 
school for school use only. (Table 2.13) 


Schools across countries varied in their implementation of policies towards different 
aspects of ICT. (Table 2.14) 


School principals had different perspectives on the priority areas for facilitating the use 
of ICT for teaching and learning (both within and across countries). (Table 2.15) 
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(Brevet Informatique et Internet), at the end of upper-secondary schooling, and aga 
nter higher education. 


ntroducing Techno 
. Forty-six schools are pa 
gn, and critical understandin 
this new curriculum. 


e national education administra 
Educat 


cal Literacy both as a subject and as material 
cipating. The curriculum includes competencies 
g of ICT. Target grade students of ICILS 2018 


ogi 
rti 


s: state 
ucation 


tion at two leve 
ion and Culture is responsible for the ed 
cation for the implementation of the policy aims 
, Municipalities are responsible for 
veness and quality of their education. 
s (e.g., recruitment of personnel). 


Edu 


the effecti 


mework of the national core 
ferent subjects and learning 
mented and assessed. ICT is 
bject based assessments (no 


ming are provided 


nthe fra 
ts of dif 


ricula withi 


part of su 


place from 2004. 


ting the new core curriculum, which has a strong focus on CIL, across all 
ple, the stated objectives of mathematics in grades 7 to 9 explicitly reference 
ing logical and algorithmic thinking, and learning good programming practices. 


ational Education and Youth is responsible for preparing the government's 
on policy and national educational curriculum. Implementation is the responsibility 
f the 30 educational districts (académies). The common base of competences (Socle commun de 


t every student must know and master 
developing the necessary skills for ICT 


ts. The latest curriculum implemented 
er code via algorithms and robotics in 
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Evaluation, Foresight and Performance assesses an 
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Germany 


Germany is a federal republic consisting of 16 federal states. Each federal state has supreme 


egislative and administra 


and qua 
evel of 


Regarding ICT integration 


and Cultural Affairs (K 


model established with IC 


This includes regulation of school curricula and pr 
ity development in schools. Inlower-secon 
CILS 2018 (grade 8), there are two to four paths of secondary education in the 
states which vary according to their respective school leaving certificates and qualifications. 


ng Conference 


s cross-federa 


tive power over all cultural policy issues including its education system. 
ofessional requirements, teacher recruitment, 
dary schooling, which includes the target grade 


federal 


into schools, the federal regulations differ between the states. However, 
n recent years, the topic of digitalization in education has moved clearly into the spotlight of 
public interest in Germany. In this context, the Stand 
Itusminister Konferenz) publi 
(Bildung in der digitalen Welt) in the year 2016. Int 
related abilities for students in prim 


of the Ministers of Education 


shed astrategy “Education inthe digital world” 
his strategy, a competence model of ICT- 
ary and secondary schools, including explicit reference to the 
LS 2018, is presented. Thi 


strategy will be carried out 


from the school year 2018/2019, after the ICILS 2018 data collection. The implementation of 


these recommendations on federa 


Furthermore, in 2016 the 


of a five billion euro budget to equ 


Federal Ministry of 


state level are still in progress 
Education and Research announced the provision 
ip schools with ICT as part of the Digitalpakt Schule project. 


in each of the federal states. 


For this initiative, the German constitution first needed to be changed, and financing schools 


wi 


Italy 
The Ministry of 


education is the 


th federal money started in 2019. 


Education, Universi 
for the public educational system. 


ational Plan for 


Di 


gital 


ty and Research issues the general guidelines and policies 
The main document establishing the objectives of digital 
Education (Piano Nazionale Scuola Digitale), organized 


into 35 broad actions covering all of the areas connected to the development of ICT in public 


education. The curriculum is derive 
one relevant to nursery schools and 
curricolo della scuola dell’infanzia e del primo ciclo di istruzione) and the other relevant to secondary 


nd its use in education 
n each subject. 


Q 


since 2018. 
Kazakhstan 


the first cycle of public educat 


and technical education (Indicazioni nazionali per i licei e Linee guida 
above mentioned documents contain specific reference to actions, 

acquiring the necessary ICT competence to complete each cycle (e.g., primary, secondary). ICT 
a is viewed as a transversal objective, necessary to fulfi 
O 


Every school has the autonomy to monitor and assess students’ progress 
ifferent tools. The Ministry has issued models for certification of competences that each 
school can use according to their own needs. In those models there is a specific entry for digital 
competences. In the first cycle of education (primary), moreover, national tests are run digitally 


d from two documents which 


provide general directives: 
ion (Indicazioni nazionali per il 


per il biennio e il triennio). The 
tools, and strategies aimed at 


the requirements 


in ICT, each using 


Kazakhstan is administratively divided into 14 regions (provinces) and three® cities of republican 


significance. 


Each region consists of several smaller districts. The Ministry of 
Science is responsible for implementation of the unified ed 
involvement of regional and district education departments. 
are standardized across the country and implemented in 


approved curricula, each school prepares its own working curr 


6 OnDecember 28, 2018, amendment to the Law of the Republic of Kaza 


Education and 


ucation and training system with an 
Education curricula and assessment 
each region and 


district. Based on 
iculum. The national school 


hstan on the administrative and territorial 


structure took place to include the third city of republican significance. During the ICILS 2018 main survey, there 


were two cities (April-May 2018). 
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system includes primary, lower secondary, and general (upper) secondary education levels. All of 
these education levels are free and guaranteed for citizens by the Constitution of the Republic 
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includes grade 
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of the adult education 
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ta 
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tate Program for Education 
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FICILS 2018, with 
tencies and increasing teacher use of ICT. 
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perate some elective courses. 


al Curriculum to deal wi 
in the target grade leve 
. These students can lear 
nformatics curriculum f 
science and fostering th 
it is an optional subject, 
s that do not. Currently, the 2015 Revised C 
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e, student achievement levels are evaluated in mos 
luated at the end of each se 


al needs. Students 
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middle school. T 
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T. However, because 
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and Research Informatio 
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g non-formal extracurricular education 
provision and support schemes. Within this mi 
for Educational and Technological Research and | 
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hool development. 
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tal media for lea 
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for primary and secondary schools. The national 
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ea 
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there are 
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uced in 2017 addressed to students in grade 11. This initiative 


emerged 
schools in 
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inthe context of a 
ICT. Learning ac 
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the learning of sci 


new label called “Future hub,’ which serves to highlight innovative high 
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ies are project-based wi 
ences and ICT, creative thinking, and communication. Learning is 


th a focus on learning autonomy. Emphasis 
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placed within a chose 
robot, game programmi 


n theme, which is cross-curricular. 


Examples include the construction of a 
ng, and the development of a website. The whole teaching and learning 


approach is based on collaboration between teachers and students; solutions are sought in 


different subjects as th 
not assessed in nation 


Portugal 


The Portuguese Mini 
and direction for sch 
schools’ resources a 


al assessments. 


stry of 
ool education conc 
nd organization. In 


Education i 


s responsible for es 
erning curriculu 


2017, the documen 


e project is implemented and completed. The use of ICT in education is 


tablishing the overarching goals 
m, national assessment, funding, and 
t “Student profile at the end of 


compulsory education,’ established the benchmark for all schools and curricula within the scope 


of compulsory educa 
digital literacy. The ICT co 
citizenship; investiga 
CT is also a mandato 


The Ministry of 
initiative “Introductio 
from grades 3 and 4, between 2015 an 
The National Network of Programm 
significant coverage in Portuguese sc 


tion. T) 


te and 
ry sub 


researc 
ject for s 


Education also promo 


nto programmi 


his included ICT and the d 
re curriculum competences a 
h; communica 
tudents from grad 


tes and suppor 


ing and Robo 
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te and colla 
e 5 to grade 9. 


opment of capacities associated with 
re organized in four domains: digital 
borate; and create and innovate. 


ts several school projects inthe area of ICT. The 
ng inthe 1st cycle of basic education” addressed to students 
d 2018, covered about half of the Portuguese school clusters. 
tics Clubs, launched in 2014/2015, achieved 
hools, with 269 clubs registered in 2018/2019. Assessment 
for ICILS 2018 target grade students is school-based and includes written assignments and 


reports, group research and investigation, oral presentations, tests, and classroom participation. 


There are no nationa 


United States 
The United S 
District of Co 
and training are 
districts), with 
ICT exist at both 
of Education 
that state. 
schools, such as setting 
and teachers usually have a high 
instructional ma 
or CT courses at the federal or s 
to incorporate ICT into other subjec 


require 


There is 
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S distri 


exams in ICT. 
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umbia. There are th 
primaril 
some decision-making o 
the federal and state 
responsible for setting p 
Districts may also set their 
technology skil 


tes (more fully, the U 
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terials, teaching 


tate leve 


CT courses, or 


no federal requi 


has been conducted as part of th 


assessments in other subjects (e.g., engine 


cts and schools generally have a 


Uruguay 
Uruguay is a South American republic with a population of 3.5m 
system. The National Administration of Public 
tate agency responsible for the planning, 
system (including preschool, primary, secondary, vocational, a 
charge of the public education system and also controls the pr 


evel of autonomy in cur 
techniques, and evaluation 


rement for assessing ICT or compu 
nost states have a compulsory assessment focused 
the federal level, a sample-based, non-compulsory assessm 


riculum deliver 


ools h 
ts. 


Is. Districts and sc 


ting-related s 
solely on IC 
ent of techn 
e National Assessment of Educ 
ering and science) may reflect so 
high level of autonomy in assess 


33820 42 


illion. There 


Education (A 


ivate system. 


dards and suggest | 
standards for various grade levels. In addi 


methods. There aren 
ave the aut 


d local. 
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tate’s 
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CT curri 


y, | 
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Departn 
truction wi 


tion, princi 
ncluding selecting 
orequired 
ity to offer and 


nited States of America) consists of 50 states and the 
evels of government: federal, state, an 
y the responsibility of the state and local governments (incl 
ccurring at the school level. Plans and policies to sup 
evels in the United States, with each s 
olicies and standards to guide school 
own academic stan 


Education 
ng sc 


hool 
port 
nent 
thin 
cula for 
pals 


CIL 


e target grade, 


at the state level. At 


ogy and engi 


neering 


ional Progress. State 
e ClLrelated aspects. 


ent of ICT ski 


is anational ed 
EP), an autonomous entit 
management, and administration of the public ed 
nd teacher education). A 
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y, is the 
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EP is in 
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There are two government institutions that aim to ensure quality of educati 


equitable education opportunities. The first is the National 
which is a fully autonomous public institution created by th 
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on and access to 
Institute for Educational Evaluation, 
e General Law of Education in 2008 


to evaluate the quality of national education. Every three years there is a national assessment 


(Aristas) of representative samples at the primary (primary 
(secondary or vocational schools at secondary grade 3). The 
and mathematics, but also examine socioemotional abilities 


grades 3 and 6) and secondary level 
assessments are focused on reading 
and school environment. 


Plan Ceibal was created in 2007 to foster inclusion and equal opportunities in schooling and 
to support the implementation of Uruguayan educational policies related to technology. Since 
it was implemented, every child in the public education system has been given a computer or 
tablet device for personal use, with free internet access at school. Plan Ceibal provides programs, 
educational resources, and teacher training courses to support the use of ICT in teaching and 
learning. More recently, Plan Ceibal has become an agent for innovation in education. In this 


new role, it aims to promote new pedagogies for learning a 
century. 


Moscow (Russian Federation) 


nd competences for the twenty-first 


The Russian Ministry of Education (Minprosveschenie) is responsible for the development and 
implementation of educational policy. Regional executive authorities (including the Department 


of Education of Moscow) are responsible for regulating ed 
exercise state control over educational activities. 


Target grade students of ICILS 2018 can be assessed in the s 


through all the learning areas) 


monitoring. Regional assessments include subject-specific 


include national Informatics exams taken by students at the 


North Rhine-Westphalia (Germany) 


and time schedules, professional requirements, teacher re 
in schools. Compulsory education begins at the age of six 


ucation within their jurisdiction and 


ubject Informatics, while ICT literacy 


is represented in the Russian curriculum as a general capability (a capability to be addressed 
The assessment of results in the Informatics subject and in 
CT literacy across subjects is carried out at the school, regional, and national levels. School 
assessments are carried out by teachers and by the administration during the school internal 


diagnostic tests on informatics and 


diagnostic assessment of general ICT literacy using computer-based testing. National assessments 


end of grade 9 and grade 11. 


North Rhine-Westphalia, with 18 million inhabitants, is the most populous of the 16 federal 
states in Germany. It has supreme legislative and administrative power over all cultural policy 
issues including its education system. This administrative power includes regulation of curricula 


cruitment, and quality development 
. In general, there are four paths of 


secondary education, which vary according to their respective school leaving certificates and 


qualifications. Compulsory schooling ends after completion 


North Rhine-Westphalia follows the cross-federal state s 
of the Ministers of Education and Cultural Affairs (Kultusm 


of lower-secondary education. 


trategy of the Standing Conference 
inister Konferenz) developed in 2016, 


which outlined general ICT competencies. In 2017, North Rhine-Westphalia (NRW) developed 
the “Media Competence Framework NRW” (Medienkompetenzrahmen NRW) which targets grade 


1 to grade 9/10. This was in place from June 2018 (immed 


iately after data collection for ICILS 


2018) and as such, the curricula of target grade students for ICILS 2018 did not explicitly cover 


|CT-related skills. In some schools and school tracks, ICT-re 


ated subjects are offered as elective 


subjects. Additionally, anumber of schools have been developing profiles in teaching and learning 


with ICT in the scope of their pedagogical autonomy. 
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Characteristics of the education systems in participating ICILS countries 
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Level of school autonomy for aspects of school policy 
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ture of school-based education also varied considerably across countries (Table 2.1 
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the International Standard Classification of 
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cation (and includes the classification of the target grade in ICILS), 
D 3 level to upper-secondary education. 
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In almost all participating educational systems at the time of the NCS data collection, schools 
reportedly had at least some autonomy for the eight different aspects. For example, for matters 
that relate to school governance (includin 
implementation of the curriculum), public schools in 12 out of 14 educational systems were 
my, whereas these schools in Moscow (Russian Federation) had 
tan were schools reported to have no autonomy for these aspects 
of governance. As a general pattern, the level of autonomy that public and private schools were 
reported to have over school policies were not the same, with private schools having a greater 
degree of autonomy. Schools in Moscow (Ru 
of autonomy out of all countries. Bo 
autonomy for all eight aspects. 


g financial management, setting strategic goals, and 


ssian Federation) reportedly have the greatest degree 


th public and private schools were reported as having full 


a greater degree of autonomy was the assessment of student 
es reported public schools have full autonomy for this aspect). 


Six out of the 14 countries or benchmarking participants reported that their public schools had 


no autonomy over the selection and appoi 
that private schools had full autonomy for 
reported to have no autonomy, or only so 
selection and purchase of ICT a 
whereas all countries with data available re 


autonomy for the purchase of ICT equipm 
autonomy for staff participation 


ntment of teachers, whereas all countries reported 
this particular aspect. Similarly, public schools were 
me autonomy, for technical support for ICT and the 


nd software in 10 of the countries or benchmarking participants, 
ported that private schools had full autonomy in these 
aspects (the exception was Luxembourg where private schools were reported as having only some 


ent). There were relatively higher reported levels of 


in professional learning inthe use of ICT (in all countries the level 


of autonomy was rated between some and full autonomy for both public and private schools). 
The level of autonomy for the assessment of student achievement in CIL (or its equivalent) was 
similarly high, with the exception of France, where it was reported that there was no autonomy 
in either public or private schools for assessment. The delivery of ICT curriculum had relatively 
lower levels of reported autonomy in comparison to other aspects of school policies. In France, 
no autonomy was given to either public or private schools for this aspect. 


ICT infrastructure and economic characteristics of countries 


to education (Table 2.3). 


An examination of the information about 


norder to provide information on antecedent aspects of national contexts for the implementation 
of ICT in education, we collected data relating to ICT infrastructure and economic characteristics 
in participating countries (Table 2.3). The ICT infrastructure data include the proportion of the 
population using the internet aged 16-74 in the last three months, and the ICT development 
index (IDI) score” and country ranking; economic development data include the gross domestic 
product (GDP), income Gini coefficient,® and the percentage of public expenditure apportioned 


infrastructure reveals diverse country profiles. The 


percentage of individuals using the internet provides an indicator of how widespread the ICT 


infrastructure is ina country. Denmark and 


Luxembourg had very high levels of access (more than 


97% of people aged 16-74 accessed the internet within three months). The lowest recorded level 


of access was in Italy (61%). 


Denmark, Fin 


land, France, Germany, Korea, Luxembourg, and the 


7 TheIDlisacomposite index that incorporates 11 different indicators relating to ICT readiness (infrastructure, access), 
ICT usage (individuals using the internet), and proxy indicators of ICT skills (adult literacy, secondary and tertiary 
enrollment). Each country is given a score out of 10 that can be used to provide a benchmarking measure to compare 
ICT development levels with other countries and within countries over time. Countries are ranked according to their 


IDI score. 


8 The Gini income coefficient is a measure of the deviation of the distribution of income (or consumption) among 
individuals or households within a country from a perfectly equal distribution. A value of O represents absolute 
equality. A value of 100 represents absolute inequality (see UNDP 2016). 
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Approaches to CIL/CT education in ICILS countries 


Details of plans and policies for the use of ICT in education 
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Inthe NCS, each country was asked a series of questions related to plans or policies that support 
the use of ICT in education. In general there was little variation across participating countries in 
their descriptions of their plans and policies. 


Support was found for all participating countries either by authorities at the local /district/ 


municipal level, at the state/provincial level, at the nationa 


level, or some combination of the 


three levels. Countries were asked to indicate whether the plans or policies explicitly or implicitly: 
emphasize different aspects improving student learning, emphasize the need for different ICT resources, 


emphasize different methods of supporting student learning, and 


include different aspects as priorities. 


Most of the following seven aspects of ICT in education were largely recognized (either explicitly 


or implicitly) across the 14 participating countries and bench 


ncreasi 
e Compu 


Respon 


The impor 
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Mainte 


in Chile); 


Preparing studen 
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t and other ICT resources; 


9 
10 


This aspect applies to a large number of German federa 
The Finnish curriculum at the time of the study did not emphasize this, but this has since been updated to have an 


emphasis on programming. 


states. 


ts) (except in 


ces? 9: and 


marking participants: 


Learning of subject matter content (art, language, mathematics, science, etc.) (except in 
Kazakhstan and the United States); 


Denmark and 


lowing ICT resources were recognized (explicitly or implicitly) 
g participants: 


ther ICT resources (except in Chile and Portugal): 


puter equipment and other ICT resources (except 


Support for teachers for using computer equipment and other ICT resources in their work; 
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e Access to digital educational resources; 


e Internet connectivity; and 


e Home access to school-based digital education resources such as through school-hosted 


e) 


nline portals (except in Chile, Kazakhstan, and Korea). 


The plans and policies of most countries and benchmarking participants emphasized the following 
methods of supporting student learning (implicitly or explicitly): 


e Pre-service teacher education in the use of ICT (except 


e |n-service teacher education in the use of ICT; 


in Portugal); 


e The use of learning management systems (except in Finland and Germany, including North 


Rhine-Westpha 


e Reporting to parents (except in Finland and Germany, including North Rhin 


e Providing feedback to stud 


and Korea). 


When asked abou 


lia); 


t the exte 


ICT, again there was near ful 


explicitly or implici 


tly mentio 


ents (except in Finland, Germany, including North 


e-Westphalia); and 
Rhine-Westphalia, 


nt that the plans and policies emphasize priori 


ned: 


e Professional development for teachers’ pedagogical use of ICT, 


e Sufficient ICT infrastructu 


e Development o 


re and resources in schools; 


f |CT-related competencies in students; 


e Development and provision of digital learning materials; 


ties for the use of 


| agreement across participating countries that the following were 


e Reduction of the digital divide between groups of students (except in Finland and Portugal); 


e Improvement of administrative and management systems in schools (except in Finland and 


France); and 
e Use of ICT 


to improve communication with parents (except in Finland and the United States). 


A reference to providing one-to-one computing in schools was noted in the plans and policies 
for Chile, Germany, Luxembourg, and Uruguay. All countries, with the exception of Kazakhstan 
and Portugal, had coverage of formal support for the development of digital resources in their 


plans and policies. 


Emphasis on aspects of CIL in plans and policies 


All countries were asked to complete a question on the extent to which their plans and policies 


for the 2018 year emphasized the following aspects of CIL in their national curricula or selected 
curricula (Table 2.4): 


e Searching for information using ICT; 


e Evaluating the reliability of information sources accessed using the internet; 


e Presenting information for a given audience or purpose using ICT. 


e Organizing information obtained from internet sources; 


e |ssues relating to intellectual property (such as copyright and attribution sources); 


e Responsible and respectful publication of information; 


e Use of productivity tools (such as word processing, spreadsheet, and presentation software); 


e |T security issues (e.g., passwords, malware, phishing); and 


e Data security (such as the collection of internet use data by search engines and social media 


sites). 
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In general, each of these aspects was included in national curriculum documents, either explicitly or 
implicitly in 12 to 14 of the participating countries and benchmarking participants. In three entities, 
Denmark, Germany, and Moscow (Russian Federation), all aspects were mentioned explicitly. The 
aspects that were most often explicitly noted in plans and policies were the use of productivity tools 
(such as word processing, spreadsheet, and presentation software) (10 countries or benchmarking 
participants) and searching for information using |CT (12 countries or benchmarking participants). 
The remaining five aspects were typically included in plans or policies, but often were indicated as 
being implicitly stated in the curriculum. 


Emphasis on aspects of CT in plans and policies 


A feature of ICILS 2018 was the international option for a student test of CT. A new question 
for the ICILS 2018 NCS was included to assess whether national curriculum emphasized aspects 
of CT. All ICILS 2018 countries were asked to indicate whether their curriculum documents 
contained each of the following aspects: 

e — Planning technology-based products or solutions; 


e Developing technology-based products or solutions to meet user requirements; 


e Evaluating and refining technology-based products or solutions; 


e Creating visual representations (e.g., flow charts and decision trees) of processes; 


e Creating visual representations (e.g., flow charts and decision trees) of information/data; 


e Designing user interfaces for technology-based products or solutions; 


e — Revising technology-based products or solutions on the basis of user feedback or other 
data; 


e Creating algorithms; 


e Writing code, programs, or macros; 
e Evaluating code, programs, or macros; 


e Developing digital applications (e.g., programs/apps); and 


e dentifying and describing the properties of digital systems. 


The extent to which CT aspects were present in curriculum documents varied across the ICILS 
2018 countries (Table 2.5). Luxembourg and Uruguay did not contain any details (explicitly 
or implicitly) of these concepts in their curriculum documents, whereas all were contained 
in documents for Denmark, Korea, the United States, and Moscow (Russian Federation). The 
e 
S 


creation of visual representations (e.g., tables, graphs, or charts) of information/data and th 
creation of algorithms were the only aspects to be explicitly stated by the majority of ICIL 
2018 participants (eight countries or benchmarking participants). Amongst the other aspects 
most likely indicated as being explicit parts of the curriculum were: writing code, programs, or 
macros (seven countries or benchmarking participants); planning technology-based products 
or solutions and the creation of visual representations (e.g., flow charts and decision trees) of 
processes (six countries or benchmarking participants). Revising technology-based products or 
solutions on the basis of user feedback or other data, identifying and describing the properties 
of digital systems, and designing user interfaces for technology-based products or solutions 
were the aspects that were least frequently suggested as being explicitly part of the curriculum. 
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Table 2.5: Emphases in national curricula of teaching aspects related to CT 


Notes: Data from the ICILS 2018 national contexts survey. 
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1 The ICILS 2018 age cohort were the last to follow the curriculum; this has been replaced with stronger emphasis on aspects of ICT including CT. 


CONTEXTS FOR EDUCATION ON CILAND CT 


School delivery and assessment of ClL-related skills 


Data from ICILS 2013 show contrasting ways in 


countries (Fraillon et a 


how Cl 


L was delivered 


2.6). Firstly, responden 


. 2014). In 
and assessed in their coun 
ts were asked whether CIL 


tries for 
was inc 


which CIL was de 
CILS 2018, each country was asked a series of 
the year o 
uded at eac 


ivered to stu 


f the data col 
h level as eith 


dents across 
questions on 
ection (Table 
er aseparate 


subject, whether it was 
integrated into other s 
the subject was compu 
have a separate subject: i 
themselves if they have sh 
States had a non-co 
exception of Kazakhstan 
science and technology s 
this was anon-comp 
level CIL was delivered i 
another subject). 


C 
benchmar 
had some 
integrated into ot 
and Uruguay (bot 
(e.g., science and 
ICILS 2018 count 
and applications d 
ruguay the incl 


N 
O 


integrated 
ubjects. For each of these 
sory or no 


mpulsory subj 
and Nort 
tudies or 
ulsory study). | 
two or 


into science and technology studies, and/or whether it was 
options they were asked to indicate whether 
-compulsory. At the primary level, it was rare for countries to 
Chile there was a compulsory subject, in Finland schools can choose 
rt compulsory or non-compulsory courses for ICT, whereas the United 
ect at this level. The remaining educational systems (with the 
h Rhine-Westphalia, Germany), had CIL skills integrated into 
integrated into other subjects (noting that in many countries 
n eight countries or benchmarking participants, at the primary 
three different ways (either as a separate subject or as part of 


L was delivered as part of a compulsory separate subject in five different countries or 
ing participants at 
sort of CIL offered a 
her studies. The excep 
h at the upper-secondary 
technology s 
ries or benchmar 
ata, although 
sion of coding and applicat 


the ISCED 1 level, and in six at the ISCED 2 level. All countries 
t the lower- and upper-secondary levels, via a separate subject or 
tions to this were in North Rhine-Westphalia (Germany) 
evel). When CIL was integrated into other subjects 
tudies), the subjects tended to be non-compulsory. In nine of the 
ing participants, a separate subject of CIL also included coding 
this was sometimes defined at the school, district, or state level. In 
ions data is at the discretion of the teachers. 


fF ICT. 
anda 
Kazakh 
were a 
suppor 


Each cou 
ted assess 
stan, Korea, Portug 
so asked whether 


U 
All. cou 
O 
m 


assessments, summative assessments, an 
entatives from Den 


Repres 
Federa 


tion) reported that 


importance of col 
of teacher experti 
and Glass 2007; Law et a 


ecting p 


Sie 
teache 


e Tech 


e Usin 


| par 
rs’ ca 


ticipati 
pacity in the fo 
capaci 


l in pedagogy; 


nical 
gICl 


ty in usi 


ntry respondents were also 
try had to indica 
ment of ICT and 


ted by ministries or departments of educati 


countries. France, Italy, Kazakhstan, Kor: 
these types of assessments 


Using previous research as a guide, the ICILS 2018 assessm 
se in ICT-related teaching and learning (Frai 


that teachers were using ICT extensively for teaching and learn 
ng countries were asked about the support and requirements for developing 


olicies regarding the assessment 
rement at school level regarding 
ade students. Only France, Italy, 
an Federation) had such policies. Respondents 
t types of ICT student assessments used or 
on, including diagnostic assessments, formative 
d national or state/provincial monitoring programs. 
mark, Finland, the United States, Uruguay, and Moscow (Russian 
all four types of assessment/monitoring were implemented in their 
ea, and Luxembourg all used between one and three of 
tries. 


s about their p 
re was a requi 
Is of target gr. 


asked question 
te whether the 
computing ski 
al, and Moscow (Russi 
there were differen 


in their coun 


Teacher support and requirements for using ICT 


ent framework highlights the 
tem level for the development 
llon et al. 2019; Charalambos 
. 2008; Scherer and Siddigq 2015). Data from ICILS 2013 confirm 
ing (Fraillon et al. 2014). In the 


rocess-related information at the sys 


lowing aspects of ICT for the year of the data collection (Table 2.7): 


ng ICT; 


e Collaboration and communication in using ICT; and 


e Using ICT 


| for student assessment. 
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CONTEXTS FOR EDUCATION ON CILAND CT 


For each aspect, respondents were asked to indicate whether learning is amandatory component 
of pre-service education, whether it was a requirement for being a teacher, and also whether 
participation in some form of professional learning program was required for teachers. The 
responses present a different profile across participating countries and benchmarking participants 
in the way in which teachers’ capacity to use ICT is mandated. In Finland, Germany (including 

orth Rhine-Westphalia), Kazakhstan, Korea, and Portugal, there are no requirements for any of 
these aspects to be learned by teachers. In Denmark, France, and Moscow (Russian Federation), 
those aspects that are required, are a mandatory component of pre-service education. In Chile, 
Luxembourg, the United States, and Uruguay, there were requirements for participation in some 
form of professional learning program. Only in Italy and the United States were teachers’ capacities 
in any of these areas a registration requirement. In both the United States and Moscow (Russian 
Federation), these aspects were all both a mandatory part of pre-service education and teachers 
were required to obtain some form of professional learning in this area. 


Support for |CT-based professional development 


The NCS also asked about the level of support and teacher access to participation in |CT-based 
professional development. Respondents were asked to judge whether any of the following 
aspects were supported by funding teacher participation in programs, by providing resources 
for teachers to access, or by providing relieving teachers to allow regular teachers to attend 
programs (Table 2.8): 


e —_Toimprove ICT/technical skills; 

e —Toimprove content knowledge with respect to CIL; 

e —Toimprove teaching skills with respect to ClL-related content; 

e To develop digital teaching and learning resources; 

e To integrate ICT in teaching and learning activities; and 

e — Toimprove skills in computer programming or developing applications for digital devices. 


All aspects were supported in various ways across all countries (the exception is improving 
skills in computer programming or developing applications for digital devices in Kazakhstan and 
Portugal). Support was more likely to be provided by funding teacher participation in programs 
and by providing resources for teachers to access, in comparison to providing relieving teachers 
to allow regular teachers to attend programs, which occurred less frequently across countries. 
There was little variation within countries over the types of supports provided across the 
different aspects. Most countries used a combination of support for each aspect. In Denmark, 
Finland, Korea, and Luxembourg, all three types of support were provided for each of the six 
aspects. In Germany, support was offered by providing resources for each of the aspects for 
teachers to access (support in North Rhine-Westphalia was provided for all except to improve 
|CT/technical skills), whereas in Kazakhstan this was most likely in the form of funding teacher 
participation in programs. 


Schools’ access to ICT resources 


Previous findings from cross-national surveys, including ICILS 2013, show differences in the 
provision of ICT resources in schools across countries (Anderson and Ainley 2010; Fraillon et al. 
2014; Pelgrum and Doornekamp 2009). School ICT coordinators were asked to identify whether 
arange of specified technology and software resources are available in their school, and whether 
these were available to only students, only teachers, or both students and teachers. 
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Table 2.7: Requirements for developing teachers’ capacity to use ICT 
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Country Technical capacity Using ICT in Collaboration and Using ICT for 
in using ICT pedagogy communication student 
in using ICT assessment 
Chile © © © © 
Denmark e e = 
Finland = 7 - = 
France e e C) e 
Germany - - - - 
taly A © © - 
azakhstan - - - 7 
orea, Republic of - - - - 
Luxembourg © © © © 
Portugal - ~ - - 
United States e A © e A© ® © e © 
Uruguay 7 © © ro 
Benchmarking participants 
oscow (Russian Federation) e © ° © ® © e © 
orth Rhine-Westphalia (Germany) - - - - 


Note: Data from the ICILS 2018 national contexts survey. 

@ Mandatory component of pre-service teacher education. 
A Requirement for registration as a teacher. 

© Participation in some form of professional learning program in this area required for teachers. 


Access to technology-based resources 


school networ 


was largely available to both grou 
almost 90 percent across countries (relative 


Email accounts 


the latter with 


tended schools 


Sli 


S. 
a 

pa 
fo 
d 


re 


CT coordinator responses on whether different technology resources were available in schools 
for both teachers and students were recorded (Table 2.9). Access to the internet through the 
ps in all participating countries, on average 
y low availability of 66% was reported for Italy). Digital 
earning resources that can only be used online were also commonly available to both teachers 
and students, on average 86 percent of students at 
On average, approximately two thirds of students across countries a 
earning resources could be accessed offline and an educationa 
education authorities was available to both teachers and studen 
evident for both types of resources, particularly for 
from Italy attending schools with these resources available, com 
of students from Moscow (Russian Federation). 
east common technology resource available for both students an 
average 55%), although it appears that these are relatively commo 
than 90% availability in Denmark, Finland, and Luxembourg), but 
availability in North Rhine-Westphalia, Germany). It is worth noti 


with this resource available. 
ttended schools where digital 
te or network maintained by 
Some country variation was 
ow of 29 percent of students 
red with a high of 94 percent 
r school-related use was the 


teachers across countries (on 
nplace insome countries (more 


atively scarce in others (21% 


ng that in a large proportion of 


schools cross-nationally, email was made available only for teachers. 
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Table 2.9: School reports on technology-related resources for both teaching 
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and learning 


Country Percentages of students at schools where technology-related resources are available for both teaching and learning 
Digital learning Digital learning Access to the Access to an education Email accounts 
resources that resources that internet through site or network for school- 
can be can only be the school maintained by related use 
accessed offline used online network education authorities 
Chile 7D: (37) 2S 87 (3.0) 85 (3.9) 60 (3.2) V 32 (6.2) W 
Denmark! * 68 (4.6) 99 (0.6) & 100 (0.0) A 87 (3.3) A 91 (2.8) 
Finland 46 (49) W 94 (2.3) A 99 (13) A 66 (4.5) 93 (2.2) 
France 78 (4.1) A B5 (3) 100 (0.0) A 79 5) A 73 (4.2) 
Germany 64 (3.9) 73 (43) VW 91 (2.6) 50 (4.1) Vv 30 (4.2) VW 
taly? 72 (37) 75 (4.0) W 66 (45) W 29 (3.8) Vv 38 (4.2) W 
azakhstan* 65 (3.8) 77 (38) V 80 (3.5) V 63 (4.1) 42 (46) W 
orea, Republic of 78 (34) A 87 (3.2) 82 (38) V 77 (4.0) A 33 (4.1) VW 
Luxembourg 61 (0) V 100 (0.0) A 100 (0.0) A 87 (0.0) A 99 (0.0) 
Portugaltt * 69 (3.4) 83 (2.7) 93 (2.0) 67 (34) 35 (34) WV 
Uruguay 74 (4.1) 86 (3.6) 86 (3.9) 87 (3.3) A 40 (54) W 
ICILS 2018 average ee (11) 86 (09) 89 (0.9) oe (1) ew uly) 
Not meeting sample participation requirements 
United States 74 (3.3) 95. (1.9) 99 (0.5) 93 (1.6) 84 (2.7) 
Benchmarking participants meeting sample participation requirements 
Moscow (Russian Federation) 83 (3.1) A 98 (0.9) A 92 (2.6) 94 (2.0) A 53 (4.7) 
North Rhine-Westphalia (Germany) 60 (4.5) 69 (44) W 86 (3.7) 40 (4.4) Vv 21 (4.5) Vv 
Notes: Standard errors appear in parentheses. Because results are rounded to the National ICILS 2018 results are: 
nearest whole number, some totals may appear inconsistent. Comparisons with A More than 10 percentage points above average 
ICILS 2018 only reported for countries or benchmarking participants meeting A Significantly above average 
sample participation requirements. ist 
— mielccee i ae ie — VY Significantly below average 
et guidelines for sampling participation rates only after replacement schools Y More than 10 percentage points below average 


were included. 


tt Nearly met guidelines for sampling participation rates after replacement schools 
were included. 


National defined population covers 90% to 95% of the national target population. 
Country surveyed target grade in the first half of the school year. 


1 


2 


Access to software resources 


ICT coordinators were also asked about the avai 
and teachers (Table 2.10). Word and presentati 
to both groups (98% on average across countr 
availability (in terms of students attending schoo 
teachers) were also found for: 


ability of so 


ies for both 
s with resou 


iting (85% on 


tware resources fo 


types of software) 
rces available to bo 


average, with natio 


e, with natio 


e Video and photo software for capturing and ed 
ranging from 66% in Italy to 96% in Finland); 
e Graphic or drawing software (76% on averag 
in Italy to 97% in Finland); 
e Digita 
37% in 
e A learn 
25% in Chile to 97% in Finland and 90% in Uruguay); 
e Practi 


r both students 


on software were almost universally available 


. High levels of 
th students and 


nal percentages 


nal percentages ranging from 48% 


contents linked with textbooks (70% on average, with national percentages ranging from 
orth Rhine-Westphalia, Germany, to 90% in Italy and Moscow, Russian Federation); 


ing management system (66% on average, with national percentages ranging from 


ce programs or apps where teachers decide which questions are asked of students (60% 


on average, with national percentages ranging from 23% in Chile to 98% in Denmark); 
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e Social media (59% on average, with national percentages ranging from 22% in France to 84% 


in Denmark); and 


e Single user digital learning games (54% on average, with national percentages ranging from 
27% in North Rhine-Westphalia, Germany, to 80% in Finland). 


The fo 
or less 


lowin 


Concept- 
in Port 


Simulatio 
8% in | 
E-portfoli 
in Uruguay); 


ulti 


and 


Data logging and mo 
3% in Italy to 46% in 


Access to technology faci 


ICT coordinators were fu 


teaching and learning of the target grade studen 


each facility was availab 
Summary percentages of 


teachers had access to the facilities (Table 2.11 


network (LAN or wifi) and 


lities 


rther asked about tech 


e either only for studen 
the proportion of respo 


) 


the use of alearning m 


average of 65% of studen 


available to both students and teachers). As with 


ts attended schools whe 


no 
ts. 


an 


ts or on 
dents w 
suggest 
agemen 


variability: access to wifl was reported as being nearly unive 


less than one fifth of students attended schoo 


S| 


n North 


Rhine-Westph 


this was available for both students and teachers. Internet-based applica 
work and space on a school network to store files were also more com 
and 58% on average respectively across countries). Both resources also 


discrepancies: availability of the former ranged from 97 percen 
percent in North Rhine-Westphalia (Germany), while availabili 
percent in Luxembourg to 19 percent in Korea. Robots or robotic devices (average 4 
intranet with applications and workplaces (average 46%), rem 
D printer (27%), were available to both target g 


(39%), anda3 
than half the time. 


t in Finlan 
ty of the 


ogy facilities available in their school 

Again, they were asked to indicate w 
y for teachers, or for both g 
ho indicated that both studen 
that access to a wireless local area 
t system were relatively common 
re these technologies were reported as bei 
the software resources, t 
rsally available 


here was con 
in Denmark, 
alia (German 


monly repor 
had consider 
d and 
atter ran 


ged 
6% 


Denma 


g software resources were available to an average of half of ICILS 2018 students 
(in terms of school availability for both teachers and students): 


mapping software (50% on average, with national percentages ranging from 25% 
ugal to 76% in Denmark); 


ns and modeling software (42% on average, with national percentages ranging from 
taly to 91% in Finland and North Rhine-Westphalia, Germany); 


os (39% on average, with national percentages ranging from 3% in Germany to 85% 


user digital learning games with graphics and enquiry tasks (29% on average, with 
national percentages ranging from 4% in North Rhine-Westphalia, Germany, to 51% in Finland); 


nitoring tools (22% on average, with national percentages ranging from 
Denmark). 


for the 
hether 
roups. 
ts and 
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ng 
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whereas 
y) where 


tions for collaborative 
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ably high 
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from 97 
,aschool 


ote access to a school network 


rade students and teachers less 
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Table 2.12: National ratios for number of students to num 


Ava 


ilability of ICT devices for students 


PREPARING FOR LIFE INA DIGITAL WORLD 


ICT coordinators at the participating schools were asked to provide information about the number 
of ICT devices that were available to students, while school principals reported the number 


of stud 


students per device (Table 2.12). Lower ratios i 


ratios indicate a school with less access to digital technologies. 


ents enrolled at their school. We used these data to compute ratios of the number of 


ndicate a better-resourced school whereas higher 


Denmark, Finland, France, and Luxembourg were better resourced countries in terms of the 


ratio of digital devices per student, these cou 


The 


countri 


ntries all had seven students per device or fewer. 


remaining countries had ratios ranging between 10 and 22 students per device. In most 


es there was a higher ratio in urban areas in comparison to rural areas, although this 


difference was only significant at the country level for Korea and Luxembourg. 


ber of ICT devices in school by school location 


Country All students By school location Difference 
Urban Rural (urban - rural) 
Chile 18 (2.6) 20 (3.3) 14 (24) 6 (3.7) 
Denmark" 5 (1.2) 7 (2.7) 3. (0.7) 4 (2.8) 
Finland 3. (0.3) (0.3) 4 1.5) -1 = (1.5) 
France 7 (0.9) 8 (1.9) 0.8) 2 (2.0) 
Germany O (06) 10 (0.7) 0.9) 2 (1.2) 
taly? 4 (1.7) 14 (2.2) 14 (2.5) O- (3.3) 
azakhstan’ 22 (1.2) 24 (1.9) 20 (2.0) 4 (3.2) 
orea, Republic of 4 (08) 14 (0.8) 0.9) 7 (1,2) 
Luxembourg 5 (0.0) 6 (0.0) 0.0) 3 (0.0) 
Portugalt** 7 (1.7) 19 (3.1) 15 (1.5) 4 (34) 
ICILS 2018 average 1 (04) 13 (0.6) OO) 3 (0.8) 
Not meeting sample participation requirements 
United States 2 (0.1) 2 (0.1) 1 (0.1) 1 (0.1) 
Benchmarking participants meeting sample participation requirements 
Moscow (Russian Federation) 13 (0.7) 13 (0.7) 
North Rhine-Westphalia (Germany) 13 (1.0) 13 (1.1) 10 (1.9) 3 (2.2) 
Notes: Data were not available for Uruguay. Standard errors appear in parentheses. Because results are rounded to the nearest whole 


number, some totals may appear inconsistent. Statistically significant (p < 0.05) differences are shown in bold. 
Met guidelines for sampling participation rates only after replacement schools were included. 


tt 


1 


2 


Nearly met guidelines for sampling participation rates after replacement schools were included. 
National defined population covers 90% to 95% of the national target population. 
Country surveyed target grade in the first half of the school year. 


School location of ICT devices and student access to portable devices 


The 
at sc 
who 
Den 
labo 
insu 
than 


hools for the teaching and learning of targe 
have portable computers under differe 
mark), the most common location of ICT d 
ratories (only a quarter of Danish students 
chalocation). The school library was the on 
half of students (on average across countri 


CT coordinator questionnaire included questions on where school ICT devices were located 


t grade students, and on the proportion of students 
nt policies (Table 2.13). In all countries (except 
evices for students of this grade was in computer 
attended schools where computers were available 
y other location that had devices available for more 
es), although this was much more commoninsome 


countries (France, Luxembourg, and Portugal in particular). In Finland (83%), Luxembourg (65%), 
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oscow (Russian Federation) (59%), and Chile (52%), class sets of computers that can be moved 
between classrooms were available to the majority of students. In 
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uay (57%), it 
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Procedures regarding different aspects of ICT 


NCS data provided evidence of how national and state/provincial plans and policies 
deliver the teaching and learning of ICT in education. In order to hel 
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ms of the percentages of 


olence) (92% on average, 
rk to all or nearly all in Germany, 


nauthorized system access or entry (91% on 
percentages ranging from 76% in Uruguay to 99% in Portugal and Moscow, 
Federation); 


ng) (87% on average, national 
and); 


ing difficulties (82% on average, 
Denmark); 


pyrig 
Russi 


hts) (80% on average, national 
an Federation); 


national percentages ranging from 


s (but during school hours) (70% 


t use of non-school related games on school computers (69% on average, national 
tages ranging from 48% in Moscow, Russian Federation, to 82% in Portugal); 


47 


“edA JOOS dU} JO JJEY ISI BY} Ul apeIS 3931} padaAins AyQUNOD 
‘Uo!}e/Ndod JaB4e} |eUIJEU BY} JO %GS 0} %OS S49A09 uO!Je/Ndod pauyap jeuoljeN  , 
“papn jou! 949M S|Ooyps JUdWATe|da4 Jaye sayes UO!JeEdIdIJed Buljdwes Joy SauljapINs yaw AeAN, 41 


z 


adesane MOjag sjulod asejuas1ad OT UeY}SIOW! & 
aBesare Mojaq AjjuUedYIUsIS A 


aBesane aroge AjJUeIYIUSIS V7 ‘papn]|dul 24am sjooyps JUdWaIe|daJ Jaqye AJUO Saje4 UO!JedIDIqed Suljduies JOJsauljapins jay), 
adesane aroge sjulod asequasJad QT uy] BOW W ‘syuaa Inbal Uo!edioiqied ajduies 8uljaaw sjuediolqed Bul ysJeWYDUAG JO SA14J}ZUNOD JO} paz4OdaJ AJUO BTOT STD YM SuOSs|4eduo7 
‘ade S}NSA BLOT S TID] |EUOHEN ‘Juays|suoou! sweadde Aew $]€}0} SWOS JAqUUINU 3jOUM 4saJeaU 9} OJ papuNos de $}NSa4 asnedagq “sasayjuaed ul weadde SjOsJa PJE PUES :S9JON 
(AUeWJa) 
(O'S) 29 | & (6%) O€ | V (ZZ) 26 | & (SE) 9S (T'S) 8¢ (Sv) 99 | VY (OO) COT] W (87) 26 | & (O'V) £7 (9v) 79 | & (6) 0% (9T) €6 eI|EYCISaAA-OUlUY YON 
V (07) 76 | A(0'S) S7 | & (97) 85 |W (ET) Z6 | W (9) 29} & (VS) Sv | V (O00) COT] W (TT) 86 | W (872) Z8 | W (OZ) 26 | W (Sv) 28 | V (L£0) 66 (UO|}eJapa  UBISSNY) MOISO|| 
sjusWasINbay uolyediaiq4ed ajdwes 8uljaau sjuedioiqsed 8urjsewysuag 
(67) 8Z (TE) 9Z (80) 86 (VT) 96 (Ve) 89 (O07) 06 (S°0) 66 (TZ) 06 (VE) VS (Ve) 69 (EC) LT (S'0) 66 S9}e4S Pe} 
sjuawasinbas uoljedioiz4ed ajdwes 8ulja9wW JON 
(TT) OZ (TT) 09 (£0) £8 (OT) 28 (ET) Tv (CT) 69 (ZO) 26 (OT) 08 (ET) €S (CU nOv Cab Ee (80) 16 asesianre BLOT SIDI 
Vv (v2) €6 | W (97) 98 (67) T6 |V (£7) 16 (T'S) Lv (6) vZ (27) S6 | A (Ev) ZZ | W (8) 69 (Ev) ZZ (E'S) 9€ | & (0'V) 9Z Aensnin 
(8'€) ZZ (Sv) 6S (EC) 88 |W (T7) 76 (Sv) 9¥ | W (67) 28 | V (50) 66 | V (V7) 06 | W (Er) v9 | Ww (VE) 08 (VE) 9% | V (£0) 66 1 e3N10q 
(TO) TZ | W (TO) CZ | V (00) 26 JV (00) 88 | A (O'O) EE (O10) 69 | V (00) 86 | A (TO) €Z | W (00) vZ | V (0'0) 62 | A (O10) S VY (00) S6 S.noquiaxn) 
A (77) 0S | & (EV) SP (O€) 88 | & (6°) 69 | & (FE) ZZ (vv) 69 (£7) 96 | W (07) Sé (67) SS (Sv) 89) W (Er) 19 | V (TZ) 96 Jo aI|Gnday ‘ea.o 
(O'v) ZZ (Tv) SS | & (GE) TP? | VY (SZ) 06 (fr) vy) & (FE) 67 | V (LO) 86 (Ee) ZZ | W (Tv) 69 | W (VE) 78} W (OE) SB (£7) €6 qUESUYEZE 
(Sv) v9 | a(S) 6v | VY (07%) 76 |W (LZ) 26 (Lv) OV} W (8€) 08 (V7) 76 | & (OV) Z9 | & (O'V) EZ | A (O'S) ZE | A (SE) 6T (£7) £8 Ale} 
(TV) 69 | & (EV) 6E (EC) T6 | & (L'7) 7S (Tv) 97 (Ev) TZ | V (O'0) COT) W (8ST) S6 | & (T'7) 9E | & (Z'¥) 09 (Ev) O€ | V (07) S6 Auewas, 
A (SY) SE | & (FE) YE (87) 98 |V (67) 28 (Cv) SE) & (SE) 9S | V (7) L6 | W (SZ) T6 | wz (EE) TZ | W (GE) 18 | & (SC) ET (97) 66 oouet+ 
Vv (SE) 88 | W (872) 68 | V (TZ) 26 | V (Z€) 68 (9) SE (27'S) 69 | A (SE) €8 (Tv) 6Z (Sv) TS (TS) S9| & (FO) T (8'Z) 06 pue|ul+ 
Vv (62) Z8 | W (ZZ) 68 | V (@7) €6 | Ww (ET) 86 (vv) vy) & (97) SS | & (87) SS (9'€) 08 (Ly) LS | A (Cv) 19 | & (EC) Z (SE) $8 7 PYeWwUEd 
(6€) 69 | A (T'S) TS | V (ST) E6 | & (E9) Sh | W (ES) O09] W (87) TB} V (80) S6 | A (87) 99 | W (9S) 19 | w (vy) 78] Ww (T9) 6S (TE) 68 a!YD 
awoy je pue (S940 10 
pue jooyps 3e /pue sjuaied) 
asn juapnys AyuNWWO0S (sanoy jooyss 
Joy sad1Aap [saiyjnoyyip Jeo0] ay} Sulinp nq) Adjua Jo 
Bujusea| sulused| 4oj jaus93UI (a2uajo1A sunoy jooyss sunoy sse|> dajndwiod eye | ssaode wa3sAs 
a}iqou ([ZuA]]Nq4aqAd] oyioads 94} J0/pue s4ajzndwod ‘Aydesszousod (s}ysi4Adoo apis}no apis}no }IS 0} pamoyje paziuoujneun 
dayjoJo/pue | “8'3) s}uapnys Jo spaau sayndwoo Jooyds uo “B'a) |eluajzew | aiemyjos “3'a) ssayndwos s4ayndwos ade s}uapnys quanaid 
Jooyps 3e 15) ssayndwod 4Jayjo spsemo} Jelsads yzIM Jooyos sawes pajejat | azelidouddeui | s}ysi4 Ajsadoud Jooyss Jooyos sunoy Jo 0} saunseow 
UMO 4134} JO do}de| jo suolAeyaq syuapnys] 0} ssa00e Jo Jooyas-uou Jo 0} ssa900e Jo Jenqoa}jaqu! Jo 0} ssao0e 0} ssa00e daquinu ay} uo Aylangas jo 
asnjuapnys | uolsiAoid ay | ajqeydeooeun doy j40ddng uoIsIAoJd ay | asn juapnys SUOI}IGIYO1g quawyyjny ayy quapn3s quapnys SUO1}911}S9y uoIs!Aoid ay. 
:pajuawajdu! a4aM | 5] JO syadse juaJaJIp MOY SuUIpsesa. $91d1]Od 3UIMO]|OJ 34} pazed1pU! SjediduLId JOOYIS a49YM SjOOYUDS je SjUapN}s JO saBeUIIIOd AayjunoD 


CONTEXTS FOR EDUCATION ON CILAND CT 


{ooyps 1D asn [5D] fo sjaadsp JUuasaffip Sulpso&as sainpadoid Jo syodas |OOYIS “PTZ a/qoL 


48 


PREPARING FOR LIFE INA DIGITAL WORLD 


e The provision of laptop computers and/or other mobile learning devices for student use at 
school and at home (61% on average, national percentages ranging from 30% in North Rhine- 


e Student access to school comp 


Westphalia, Germany, to 89% in Denmark and Finland); and 


percentages ranging from 21% in France to 87% in Moscow, Russian Federation). 


uters outside school hours (53% on average, national 


Aspects that were less commonly identified across countries (less than half of students attended 


schools with these policies) include: 


e The provision of access to school co 
and/or others) (41% on average, n 


Moscow, Russian Federation); and 


mputers and/or the internet for the local community (parents 
ational percentages ranging from 22% in Korea to 67% in 


e Restrictions onthe number of hours students are allowed to sit at a computer (31% on average, 
national percentages ranging from 1% in Finland to 85% in Kazakhstan). 


Priorities for facilitating ICT in teaching and learning 


Principals were also asked to rate the priority (“high priority; “medium priority,’ “low priority,’ “not 
a priority”) in their school for methods of facilitating |CT use in teaching and learning. 
given a high priority (Table 2.15) by countries included: 


ncreas 


ncreas 


ncreas 
froma 


Other me 


for less than half of students (on average across count 


Suppor 


ranging from a low of 29% in 


Increasi 


Increasi 
average, ranging froma 
th 
ranging from a low of 24% in Finland to a high of 92% 


Providi 


ncreas 


for Luxe 


average, 
Federati 


ranging 


on); and 


ng the availabi 


ng teachers wi 


ing the professi 


Establishing or enhan 


from alow of 20 


ting participation in professional development on 
Denmark to a high of 87% in Moscow, Russian Federat 
ng the numbers of computers per student in th 
alow of 24% in Luxembourg to a high of 69% in Kazakhstan); 


Oo 


ing the range of digital learning resources available for teaching and lear 
all countries ranged between 40% in Denmark to a high of 78% in Moscow, Russian 


thods of ICT use that were reported as being a 
ries) included: 


ity of qualified technical personnel to support the use of ICT 
ow of 25% in Korea to a high of 69% in Moscow, Russian Federation); 


nal lea 


. 


from alow of 21% in F 
Federation); and 


n Moscow, Russian Federation). 


medium or high priority in th 


pedagogical use of ICT (48% 0 


e school (46% on average, ran 


Methods 


ing the bandwidth of internet access for the computers connected to the internet (59% 
on average, this typically ranged between 60% and 78%, with considerably lower proportions 
mbourg, Korea, and Denmark); 


ning (55% on 


ing the number of computers connected to the internet (53% on average, ranging 
ow of 23% in Luxembourg to a high of 72% i 


e schools 


naverage, 
ion); 


ging from 


T (45% on 


incentives to integrate ICT use in their teaching (41% on average, 
in Moscow, Russian Federatio 


n); 


rning resources for teachers in the use of ICT (40% on average, 
% in Luxembourg to a high of 69% in Kazakhstan); 


ng an online learning support platform (37% on average, ranging 
rance, Germany, and Luxembourg to a high of 64% in Moscow, Russian 


Providing more time for teachers to prepare lessons in which ICT is used (24% on average, 
ranging from alow of 6% in Denmark to a high of 59% in Kazakhstan). 
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CHAPTER 3: 


Students’ computer and information 
literacy 


Chapter highlights 


Computer and information literacy (CIL) achievement can be described across four levels 
of increasing sophistication. 


e Students working at Level 1 demonstrate a functional working knowledge of computers 
s tools. (Table 3.2) 


al 

Students working at Level 2 use computers, under direct instruction, to complete basic 
and explicit information gathering and management tasks. (Table 3.2) 
S 
Cc 


tudents working at Level 3 demonstrate the capacity to work independently when using 
omputers as information gathering and management tools. (Table 3.2) 


e Students working at Level 4 execute control and evaluative judgment when searching 
for information and creating information products. (Table 3.2) 


Students’ CIL varied more within countries than across countries. 


e The range between the lowest five percent and the highest 95 percent of students’ CIL 
scores within countries varied between 216 scale points (in Denmark) and 347 scale 
points (in Kazakhstan). (Table 3.4) 


e The difference between the highest and lowest average CIL scores across countries was 
157 scale points. (Table 3.4) 


ClL achievement was associated with student gender. 
e Female students demonstrated higher CIL achievement than male students. (Table 3.7) 


e The average CIL scores of female students was statistically significantly higher than that 
of male students in 10 of 13 countries and benchmarking participants that met the ICILS 
technical requirements. (Table 3.7) 


Socioeconomic status (SES), denoted by parental occupation, parental education, and number 

of books in the home, was significantly positively associated with student CIL achievement. 

e In all countries, students in the high SES groups scored significantly higher than those in 
the lower SES groups on the CIL achievement scale. (Table 3.8) 


Immigrant background and language background were associated with student CIL. 


e In nine of 13 countries and benchmarking participants that met the ICILS technical 
requirements, students from non-immigrant families had statistically significantly higher 
ClL scores than students from immigrant families. (Table 3.9) 


e In 10 of 13 countries and benchmarking participants that met the ICILS technical 
requirements, students who reported mainly speaking the language of the ICILS test 
at home had statistically significantly higher CIL scale scores than those who reported 
speaking another language at home. (Table 3.9) 
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Access to computers at home and years’ experience using computers were associated with 


students’ CIL. 


e Inall countries and benchmarking participants that met the ICILS technical requirements, 
students who reported having two or more computers at home had statistically significantly 
higher ClL scores than students who reported having fewer than two computers at home. 


(Table 3.10) 


e In 12 of 13 countries and benchmarking participants that met the ICILS technical 


requirements, students who r 


eported having five years or more experience using 


computers had statistically significantly higher CIL scale scores than those who reported 
having less than five years’ experience. (Table 3.10) 


STUDENTS’ COMPUTER AND INFORMATION LITERACY 


Introduction 


The International Computer and 
informati 


defines computer and 


Information Literacy Study (ICILS) 2018 assessment framework 
on literacy (CIL) as an “individual’s ability to use computers to 


investigate, create, and communicate in order to participate effectively at home, at school, in the 


workplace, and in society” (Frail 


on students’ abilities 


In this chapter, we de 
across ICILS countries. 


to use con 
produce and exchange informa 
each of which is specified in term 


tail the meas 


2019). 


focuses on the ex 


nstrument and data. We a 
inthis c 


onet al. 2013, p. 17). In ICILS, there is an operational emphasis 
nputer technologies to collect and manage information, and to 
tion. According to the framework, CIL comprises four strands, 
s of anumber of aspects. The strands describe CIL in terms 
of the following: understanding computer use, gathering information, producing information, and 
digital communication (Fraillon et al. 


urement of CIL in ICILS and discuss student achievement 
We begin the chapter by describing the CIL assessmen 
the proficiency scale derived from the ICILS test i 
discuss the student test results relating to CIL. The majority of content 
Research Question CIL 41, which 


t instrument and 
so describe and 
hapter relates to 


tent of variation existing among and within 


countries with respect to student CIL. In the final sections of the chapter we address aspects of 
Research Question CIL 3 focusing on the relationships between students’ levels of access to, 


familiarity with, and self-reported proficiency in usi 
focuses on aspects of stude 


Question CIL 4 which 


ng computers and their C 
nts’ personal and social ba 


as gender and socioeconomic background) and their CIL. 


Assessing CIL 


L, and Research 
ckgrounds (such 


The ICILS assessment design was established for the first cycle of ICILS in 2013. The design 
built on existing work in the assessment of digital literacy (Binkley et al. 2012; 
ICT literacy (ACARA 2012). It also included the following essential features of assessment in 


this domain: 


e Students completing tasks solely on computer; 


e The tas 
e Thetas 
e Thetas 


Inorder to ensure standardization of stude 


data, the ICILS instrumen 


s combining technical, receptive, producti 


s having a real-world, cross-curricular focus; 


s referencing safe and ethical use of computer-based information. 


create in an authentic environment without the comparability of student data 


contaminated by differen 
environment. 


The ICILS 2018 test inst 


to complete. Three of the 


tial exposure tod 


were included to enable d 


est 


the 


rument was built to be co 
ICILS 2013 and comprised five modules of questions and tasks which too 


igital resources and informati 


Dede 2009) and 


ve, and evaluative skills; and 


nts’ test experiences and comparability of the resultant 


t operates in a “walled garden,’ which means students can explore and 


being potentially 


on from outside the test 


nsistent with the instrument developed for 


the CIL 
over time 


for ICILS 2018. T 


the content of 
the existing content of 
ter-based applications 
om the set of five in a 
cluding the computer- 


30 minutes each 
modules were secure modules from ICILS 2013 (tre 
ata collected in ICILS 2018 to be reported on 
established as part of ICILS 2013 and to compare CIL achievement 
participated in both cycles. Two new CIL test modules were developed 
modules were developed to be consistent with the overarching design and conc 

ablished for use in ICILS 2013. They were also developed to represent 
2018 assessment framework and used contexts that both complemented 
CILS trend modules and reflected changes in student use of compu 
since 2013. Each student completed two modules randomly allocated fr: 
complete balanced rotation. Full details of the ICILS assessment design, in 
based test interface, can be found in the ICILS assessment framework (Fraillon et al. 


nd modules) and 
proficiency scale 
in countries that 
he new 
eptual principles 
the ICILS 
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Each CIL test module is comprised of a set of questions and tasks based on a real-world theme 
and following a linear narrative structure. Each module has a series of smaller discrete tasks," 
each of which typically takes less than a minute to complete. The narrative of each module 
positions the smaller discrete tasks as a mix of skill execution and information management tasks 
that students need to do in preparation for completion of a large task. The large task in each 
module typically takes 15 to 20 minutes to complete. Students are free to control the time they 
take to complete each task, however, in each module they are given an indication of how much 
time is recommended for them to leave available to complete the large task. 


When beginning each module, students were presented with an overview of the theme and 
purpose of the tasks in the module, as well as a basic description of what the large task would 
comprise. Students were required to complete the tasks in the allocated sequence and could 
not return to review completed tasks. There were five ICILS assessment modules and large 
tasks (Table 3.1). 


Table 3.1: Summary of ICILS CIL test modules and large tasks 


Module Description and large task 


Band competition Students plan a website, edit an image, and use a simple website builder to create a 
webpage with information about a school band competition. 


Breathing Students manage files and evaluate and collect information to create a presentation 
o explain the process of breathing to eight- or nine-year-old students. 


School trip Students help plan a school trip using online database tools and select and adapt 
information to produce an information sheet about the trip for their peers. The 
information sheet includes a map created using an online mapping tool. 


Board games Students use a school-based social network for direct messaging and group posting 
o encourage peers to join a board games interest group. 


Recycling Students access and evaluate information from a video sharing website to identify a 
suitable information source relating to waste reduction, reuse, and recycling. Students 
ake research notes from the video and use their notes as the basis for designing an 
infographic to raise awareness about waste reduction, reuse, and recycling. 


Data collected from the five test modules were used to measure and describe CIL in this report. 
In total, the data comprised 102 score points derived from 81 discrete questions and tasks. 
Just over half of the score points were derived from criteria associated with the five large tasks. 
Students’ responses to these tasks were scored in each country by trained expert scorers. Data 
were only included where they met or exceeded IEA technical requirements. The ICILS 2018 
technical report (Fraillon et al. 2020) provides further information on adjudication of the test data. 


As noted previously, the ICILS assessment framework has four strands, each specified in terms 
of several aspects. The strands refer to the overarching conceptual category for framing the skills 
and knowledge addressed by the CIL instruments, while the aspects further articulate CILin terms 
of the main (but not exclusive) constituent processes that underpin the skills and knowledge. We 
used this structure primarily as an organizational tool when describing the breadth of content 
of the CIL construct. The structure was not intended to form the basis of analysis and reporting 
of achievement by sub-dimensions (such as by strand or aspect). 


11 These tasks can be described as discrete because, although connected by the common narrative, students completed 
each one sequentially without explicit reference to the other tasks. 
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Table 3.2: CIL described achievement scale 


Description of the proficiency level 


Examples of tasks achieved by students at this 
proficiency level 


Level 1 (from 407 to 491 scale points) 


Students working at Level 1 demonstrate a functional 
working knowledge of computers as tools and a basic 
understanding of the consequences of computers 
being accessed by multiple users. They apply 
conventional software commands to perform basic 
research and communication tasks and add simple 
content to information products. They demonstrate 
familiarity with the basic layout conventions of 


electronic documents. 


Students working at Level 1, for example: 

¢ Open alink in anew browser tab 

e Use an appropriate communication tool for a 
particular communicative context 

e Identify who receives an email by carbon copy (CC) 


e Identify problems that can result from mass 
messaging 

e Record key points from a video into a text-based 
note taking application 


e Use software to crop an image 
e Place atitle in a prominent position on a webpage 


¢ Create a suitable title for a slide show 


¢ Demonstrate basic control of color when adding 
content to a simple document 


e Insert an image into a document 

e Suggest one or more risks of failing to log out from 
a user account when using a publicly accessible 
computer 


Level 2 (from 492 to 576 scale points) 


Students working at Level 2 use computers to 
complete basic and explicit information gathering 
and management tasks. They locate explicit 
information from within given electronic sources. 
These students make basic edits and add content to 
existing information products in response to specific 
instructions. They create simple information products 
hat show consistency of design and adherence to 
ayout conventions. Students working at Level 2 
demonstrate awareness of mechanisms for protecting 
personal information and some consequences of public 
access to personal information. 


Students working at Level 2, for example: 

e Add contacts to a collaborative workspace 

e Explain the advantages of using a communication tool 
for a particular communicative context 


Explain a potential problem if a personal email 
address is publicly available 

Associate the breadth of a character set with the 
strength of a password 


e Navigate to a URL presented as plain text 

e Insert information to a specified cell in aspreadsheet 
e Locate explicitly stated simple information within a 
website with multiple webpages 

¢ Know that search engines can prioritize sponsored 
content over non-sponsored content 


¢ Differentiate between paid and non-paid search 
results returned by a search engine 

e Explain a benefit of citing sources of information 
obtained from the internet 
e Use formatting and location to denote the role of a 
itle in an information sheet 


e Use the full canvas when laying out a poster 


e Control the size of elements relative to one another 
when laying out a poster 

e¢ Demonstrate basic control of text layout and color 
use when creating a slide show 


e Use asimple webpage editor to add specified text to 
a webpage 


Table 3.2: CIL described achievement scale (contd.) 
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Description of the proficiency level 


Examples of tasks achieved by students at this 
proficiency level 


Level 3 (from 577 to 661 scale points) 


Students working at Level 3 demonstrate the capacity 
to work independently when using computers as 
information gathering and management tools. These 
students select the most appropriate information 
source to meet a specified purpose, retrieve information 
from given electronic sources to answer concrete 
questions, and follow instructions to use conventionally 
recognized software commands to edit, add content to, 
and reformat information products. They recognize 
that the credibility of web-based information can be 
influenced by the identity, expertise, and motives of the 
creators of the information. 


Students working at Level 3, for example: 

¢ Identify that a generic greeting in an email suggests 
hat the sender does not know the recipient 

e Explain the disadvantages of using acommunication 
ool for a particular communicative context 

e Evaluate the reliability of information presented on 
acrowdsourced website 
e Identify when content published on the internet 
may be biased as a result of a publisher’s content 
guidelines or advertising revenue directing content 


¢ Explainthe purpose of explicitly labelling sponsored 
content published on the internet websites 


¢ Select relevant information according to given 
criteria to include in a website 

e Explain the benefit of a common information 

organization and retrieval system 

e Know what information is useful to include when 

recording a source of information from the internet 


e Use generic online mapping software to represent 
ext information as a map route 

¢ Select an appropriate website navigation structure 
or given content 
e Select and adapt some relevant information from 
given sources when creating a poster 

¢ Demonstrate control of image layout when creating 
a poster 

e Demonstrate control of color and contrast to 
support readability of a poster 

¢ Demonstrate control of text layout when creating a 
presentation 


Level 4 (Above 661 scale points) 


Students working at Level 4 select the most relevant 
information to use for communicative purposes. 
They evaluate usefulness of information based 
on criteria associated with need and evaluate the 
reliability of information based on its content and 
probable origin. These students create information 
products that demonstrate a consideration of 
audience and communicative purpose. They also use 
appropriate software features to restructure and 
present information in a manner that is consistent 
with presentation conventions. They then adapt 
hat information to suit the needs of an audience. 
Students working at Level 4 demonstrate awareness of 
problems that can arise regarding the use of proprietary 
information on the internet. 


Students working at Level 4, for example: 

e Evaluate the reliability of information intended to 

promote a product on a commercial website 

¢ Select and use relevant images to represent a 

hree-stage process in a presentation 

¢ Select and use relevant images to support 

information presented in a digital poster 

e Select from sources and adapt text for a presentation 

so that it suits a specified audience and purpose 

¢ Demonstrate control of color to support the 

communicative purpose of a presentation 

e Use text layout and formatting features to denote 

he role of elements in an information poster 

¢ Create a balanced layout of text and images for an 

information sheet 

¢ Recognize the difference between legal, technical, 
and social requirements when using images on a 
website 

e Explain that passwords can be encrypted and 
decrypted 
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Table 3.2: CIL described achievement scale (contd.) 


Description of the proficiency level 


profici 


ency level 


Examples of tasks achieved by students at this 


Level 4 (Above 661 scale points) 


¢ Source relevant facts from electronic sources for 
use in asocial media post to generate support 

e Explain how communication tools can be used to 
demonstrate inclusive behavior 

e Cite the relevant source of information from the 
internet when constructing an information product 


Describing CIL learning progress 
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Inthis section we briefly describe the key characteristics of each level onthe CIL scale with a focus 
on the differences between achievements at each level. These differences are discussed with a 
gideas for educators about target areas for teaching to support students’ learning 
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Students working at Level 4 execute control and evaluative judgment when searching for 
information and creating information products. They also demonstrate awareness of audience 
and purpose when searching for information, selecting information to include in information 
products, and formatting and laying out the information products they create. Students working 
at Level 4 additionally demonstrate awareness of the potential for information to be acommercial 
and malleable commodity and apply the conventions of a given communication tool in a given 
context to support inclusivity. 


Example CIL items 


To provide a clearer understanding of the nature of the scale items, we include i 
the chapter a set of example items. These indicate the types and range of items that students 
were required to complete during the ICILS test o 
responses corresponding to the different proficiency 


The example items are all from the band competition 
work on a sequence of tasks associated with plannin 
Students were then asked to create a website page to represent one of th 
competition. In this section we present 
competition's large task and a discussion of its scoring criteria. Th 
large task criteria illustrate achievemen 
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The item assessed students 


owing a browser or web application to save a password 
ng a computer that could be accessed by other people. Students who referred to 
webmail account or access to private information stored in the 
this item. On average across all countries, 64 percent of students 
The percentages across countries and benchmarking 


Example Item 2 (Figure 3.2) required students to explain how the characteristics of a password 
can improve the secureness of the password. 


Students were presented with two passwords and asked to choose the most secure and explain 
their choice. Student responses were scored as correct if they selected the password Fky_38% 
and included an explanation that related the broader character set used in the second password 
to password security. Acorrect response to this item illustrates achievement at Level 2 of the CIL 
scale. On average across all countries, 62 percent of students achieved full credit on this item. 
The percentages across countries and benchmarking participants varied from 27 to 80 percent. 


12 Allscorers across countries were provided the same set of example responses as the basis for training. 

13 Three hundred student responses to each constructed response item and large task criterion were independently 
scored by two scorers in each country in order to assess the reliability of scoring of each item or task within each 
country. The only data included in the analysis were those with a scoring reliability of at least 70 percent. 
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Figure 3.1: Example Item 1 with framework references and overall percent correct 


@ [School Name] [Webmail] —-/x Time 
| File Edit Tools Remaining 
| Z _ hite:iwww [schooiname] icils/[webmail] > 00 mins 


| [School Name] [Webmail] || 


[School Name] [Webmail] “=. 


Remember my password 


Username: (eal 
Password: [ea 


4 = Q [School Name} [We 


You are accessing your school's webmail service on a shared computer in a computer room. 


What is one risk to you if you tick the ‘Remember my password’ box? 


ClL scale level CIL scale difficulty ICILS 2018 average percentage correct 
responses 
1 489 64 (0.5) 


Item descriptor 


Identify a danger of ticking “Remember your password” on a shared computer 


ICILS assessment framework reference 


42 Digital communication 


Using information responsibly and safely 


62 
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Figure 3.1: Example Item 1 with framework references and overall percent correct (contd.) 


Country Percentage of correct responses 
Chile 64 (1.7) 
Denmark? 72 (1.3) 
Finland 70 (1.7) 
France 51 (1.5) 
Germany 56 (1.3) 
Kazakhstan! 50 (2.2) 
Korea, Republic of 77 (1.4) 
Luxembourg 56 (0.9) 
Portugaltt? 84 (1.2) 
Uruguay 9 (2.1) 
Testing at the beginning of the school year 
taly 35 (1.7 
Not meeting sample participation requirements 

United States 58 (1.1 
Benchmarking participants meeting sample participation requirements 

Moscow (Russian Federation) 71 (1.8 

North Rhine-Westphalia (Germany) 57 (2.0 


Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some 


totals may appear inconsistent. 


1 Met guidelines for sampling participation rates only after replacement schools were included. 
1 Nearly met guidelines for sampling participation rates after replacement schools were included. 
+ National defined population covers 90% to 95% of national target population. 


Figure 3.2: Example Item 2 with framework references and overall percent correct 


@ [School Name] [Webmail] 


File Edit Tools 


Z _ hito:worw [schooiname] icils/[webmail] 


[School Name] [Webmail] | 


| [School Name] [Webmail] ie “. o 


Username: 


Password: 


Remember my password 


=" Time 
Remaining 


> 00 mins 


# = @ [School Name] [We 


Your [webmail] account needs a password to access emails. Which password is more secure? 


© fky_345 © Fky_38% 
Select one password and explain your answer. 


[# POCO 
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Figure 3.2: Example Item 2 with framework references and overall percent correct (contd.) 


ClL scale level ClL scale difficulty ICILS 2018 average percentage correct 
responses 
2 493 62 (0.5) 


Item descriptor 


Explain the characteristics that make one of two passwords more secure 


ICILS assessment framework reference 


LA Understanding computer use 


Foundations of computer use 


Country Percentage of correct responses 
Chile 56 (1.7) 
Denmark‘? 77(15) 
Finland 78 (1.4) 
France 61 (1.8) 
Germany 79 (1.5) 
azakhstan* 27 (1.8) 
orea, Republic of 43 (1.7) 
Luxembourg 74 (0.8) 
Portugal't+ 68 (1.6) 
Uruguay 56 (1.9) 
Testing at the beginning of the school year 
taly 49 (1.5) 
Not meeting sample participation requirements 
United States 71(1.1) 
Benchmarking participants meeting sample participation requirements 
Moscow (Russian Federation) 65 (1.8) 
North Rhine-Westphalia (Germany) 80 (1.3) 


Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some 
totals may appear inconsistent. 


+ Met guidelines for sampling participation rates only after replacement schools were included. 
tt Nearly met guidelines for sampling participation rates after replacement schools were included. 
1 National defined population covers 90% to 95% of national target population. 
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Example Item 3 (Figure 3.3) illustrates student achievement at Level 3 on the CIL scale. It was 
the fourth task in the narrative sequence of the module and presented the students with four 
diagrams that represented website structure templates for the band competition website. Each 
template could be viewed by clicking the template tabs above the diagram. 


The page content boxes represented the webpages that comprise the band competition website. 
Students could arrange the page content onto the templates to evaluate the suitability of 
each template. Each template page had its own set of content boxes which could be arranged 
independently. 


Students that selected Template 3 received credit for this item. On average across all countries, 
30 percent of students achieved full credit on this item. The percentages across countries and 
benchmarking participants varied from 23 to 44 percent. 


Figure 3.3: Example Item 3 with framework references and overall percent correct 


Time 
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Figure 3.3: Example Item 3 with framework references and overall percent correct (contd.) 


ClL scale level 


ClL scale difficulty 


ICILS 2018 average percentage correct 
responses 


3 


631 


30 (0.5) 


Item descriptor 


Compare four website navigation structures and select the most appropriate for given webpage content 


ICILS assessment framework reference 


D2: Gathering information 


Managing information 


Country Percentage of correct responses 
Chile 28 (1.7 
Denmark? 34 (2.2 
Finland 32 (1.6 
France 28 (1.3 
Germany 29 (1.4 
azakhstan 25 (1.8 
orea, Republic of 35.(1.7 
Luxembourg 27 (0.8 
Portugaltt* 36 (1.6 
Uruguay 24 (1.7 
Testing at the beginning of the school year 
taly 27 (1.5 
Not meeting sample participation requirements 
United States 29 (1.0 
Benchmarking participants meeting sample participation requirements 
Moscow (Russian Federation) 44 (1.8 
North Rhine-Westphalia (Germany) 23 (1.5) 


Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some 


totals may appear inconsistent. 


1 Met guidelines for sampling participation rates only after replacement schools were included. 
tt Nearly met guidelines for sampling participation rates after replacement schools were included. 
1 National defined population covers 90% to 95% of national target population. 
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Example Item 4 (Figure 3.4) required students to evaluate different issues relating to the publishing 
of an image on a website. Each of the five issues presented pertained to one of three aspect of 
content publishing: legal, technical, and social/personal. Students could drag the issues presented 
in the boxes into the columns to show their answer. 


Students received partial credit (one from a possible two score points) if they correctly classified 
four of the five issues. This level of credit was located at Level 2 on the CIL scale. Students 
received full credit (two from a possible two score points) if they correctly classified all five of 
the issues. This level of credit was located at the boundary between Levels 3 and 4 on the CIL 
scale. On average across all countries, 62 percent of students achieved a score of at least one 
(.e., partial or full credit) on this item. The percentages of students achieving a score of at least 
one across countries and benchmarking participants varied from 37 to 83 percent. On average 
across all countries, 21 percent of students achieved full credit on this item. The percentages 
across countries and benchmarking participants varied from 10 to 35 percent. 


Figure 3.4: Example Item 4 with framework references and overall percent correct 


Legal requirements Technical requirements Social/personal requirements Issues Time 
Remaining 
Are you sllowed to edit the image? : hoo ' 00 mins 
H : i H Is the resolution appropriate for the 


web? 


Is the file format appropriate for the 
web? 


What ere the restrictions on who is 
allowed to use the image? 


Do your website partners like the 
image? 


Who owns the image? 


Click on > when you have completed the task. 


There are a number of issues you need to consider when placing an image on a website. 
Drag and drop (move) the labels above to match the issues with the requirements they fall under. One has been done for you. 
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Figure 3.4: Example Item 4 with framework references and overall percent correct (contd.) 


Score ClL scale level ClL scale difficulty ICILS 2018 average 
percentage correct responses 

At least one of two points 2 502 62 (1.5) 

Two points 4 661 21 (0.4) 


Item descriptor (one out of two scale points) 


Recognize four of five distinct legal, technical, and social issues associated with image use on a website 


Item descriptor (two out of two scale points) 


Recognize five distinct legal, technical, and social issues associated with image use on a website 


ICILS assessment framework reference 


4.2 Digital communication 


Using information responsibly and safely 


Country Percentage scoring at least Percentage scoring 
one out of two points two out of two points 
Chile 52 (2.0) 12 (1.4) 
Denmark?! 80 (1.6) 27 (1.7) 
Finland 77 (1.7) 35 (1.8) 
France 58 (1.5) 21.(1.3) 
Germany 71 (1.8) 28 (1.6) 
azakhstan! 37 (1.9) 12 (1.1) 
orea, Republic of 83 (1.4) 25 (1.2) 
Luxembourg 55 (1.1) 21 (0.6) 
Portugaltt+ 62 (1.6) 24 (1.5) 
Uruguay 46 (2.0) 10 (1.1) 
Testing at the beginning of the school year 
taly AO (1.6) 16 (1.2) 
Not meeting sample participation requirements 
United States 51(1.1) 20 (1.0) 


Benchmarking participants meeting sample participation requirements 


Moscow (Russian Federation) 70 (1.7) 34 (1.6) 


North Rhine-Westphalia (Germany) 71 (1.6) 28 (1.6) 


Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some 
totals may appear inconsistent. 


1 Met guidelines for sampling participation rates only after replacement schools were included. 
* Nearly met guidelines for sampling participation rates after replacement schools were included. 
1 National defined population covers 90% to 95% of national target population. 
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Figure 3.5: Band competition: large task details 


LARGE TASK DETAILS 


Click on [29 to review the assessment criteria. 


Before you begin this task you will watch a demonstration of how to use the software. 


Create a new profile page for the band [Band Name]. Use the instructions [Female Name 1] emailed you. 


Time 
Remaining 


00 mins 


Click on > to watch the demonstration. 


Figure 3.6: Band competition: assessment criteria review 


+ attention to the instructions 
+ layout of the text 
« layout of the images 


« organization of the page content. 


The assessment criteria for this task are: 


Create a new profile page for the band [Band Name]. 


Use the instructions in the email. 


Click on |, | to review the assessment criteria. 
Click on > when you have completed the task. 


, 
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Figure 3.7: Band competition: large task webpage editor software 


@ [Web Editor] 


File Edit Tools 


& | hite:twebeditor iclstand-competition/ 


| [School Name] [Webmail] | [eb Editor] | + 


% = @ Med Editor 


Create a new profile page for the band [Band Name]. 


Use the instructions in the email. 


Click on | Q. | to review the assessment criteria. 


Click on > when you have completed the task. oy 


Figure 3.8: Band competition: large task instruction email 


@ [School Name] [Webmail] 
File Edit Tools 
> | tp:nww {schooIname|icis/webmaljinb ‘> 00 mins 
[School Name] [Webmail] “= 
@ From: [Female Name 1] Reply “Reply toall Forward Delete [mal 
om li! = 
Gsm eal 
= Subject: Profile Page 
Q Drafts HL = fei 
Thanks for creating a new profile page for the band [Band Name]. [ea 
@ Sent This is what you must do: 
1. Add the band name to the page. [| 
@ Deleted 
2. Add the band group photo. 
3. Add the band competition logo. 
4. Add the Band Description text below. 
Band Description 
[Band Name] is a modem rock band with a twist of jazz. Their energetic singer gets the crowd moving and their 
rock sound gets them singing. The band members are [Male First Name 3 & Last Name 1] (lead singer). 
[Female First Name 3 & Last Name 3] (guitar) and [Male First Name 4 & Last Name 2] (percussion). 
Thanks a lot! 
# = @ [School Name] [We 


Create a new profile page for the band [Band Name]. 


Use the instructions in the email. 


Click on 1Q. | to review the assessment criteria. 
Click on > when you have completed the task. 
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When students had completed their webpage, they clicked on the “I’ve finished” button, an 
action which saved their webpage as the “final” version. (The test delivery system also completed 
periodic automatic saves as a backup while students were working on their tasks.) Students 
then had the option of exiting the module or returning to the large task to continue working. 


Once students had exited 


for later scori 
a set of 


were only inc 


The large tasks 
these fell into two 
to technical prot 
of color across t 


Assessment of 
e lower end, 


needs, selecting i 


structuring the in 
ther zero (no cr 


el 
as zero (no credit 
The 


each 
ina 


task. For exa 
presentation 


slides. The band com 
e necessary ele 


to th 


The 
leve 
refe 
each criterion, 
countries (Tab 


scoring cri 


sonthe CILs 


an 


Th 
by 
range to dem 
levels shown 


e design of 


uded in ana 


ficiency 
he tasks. 


technica 
to the use of 
e work at the higher end. The criteria t 
e€ purpose of communi 
formation managem 


manifestation 


teria 
rences, relevant 


e 3.3) 
each criterion appea 


the la 


categori 


en 


nform 


it 
ne 


ed 
,O 
at 
mple, 
with 
peti 


used 
eand 
score 
the mi 
(Fu 


Ca 


d 


the module the final version of the webpage was saved in preparation 


CS: 


Usuda 


yre 


prof 


cien 
the 


tcen 
ation relevant to 
formation within the ta 
or one (full credit) 


partial credit), or 


the asses 
inform 
multi 


p 


CIL 


nim 


Nd 


le slide 


tion large task com 
ments a 


conte 


for the band 


scaled 


category a 
imum and 


details of 


rinA 


te dif 


Criteria 4, 5, 6, and 7 eac 


as zero o 
criteria (1 


r one); the descri 
Table 3.3). Criter 


ppe 


rge tasks 
students regardless of the 
onstra 
in the scoring cr 


pt 


ndix B. 
in the 


ir level of profic 


ion correspondi 


cy typi 


ng by trained scorers within each country. 
seven criteria. As was the case for the open res 
yses if they met | 


in the ICILS test modules were all scored 
technic 
ated to elements such 


al profi 


cation rather than on simply an execution 
tered on elements such as adapting 
the task (or omi 
sk. Some criteri 
score points; o 
two (full credit) score poin 


sment criteria across the dif 
ation flow or consistency of 
ires considera 
prised a webpage. As suc 


s requ 


nt of awebpage. 


com 
ifficul 
nd maximum sco 
maxi 


the pe 


rcentages 


ICILS assessmen 


EA standards 


iency. Th 


ponse items 


Each webpage was scored according to 


using task- 


a allowed 


ferent tas 
formattin 


8 


n, 


petition’s large task are presented according to th 
ties as wellas their ICILS 2018 assessment framework 
re, the percentage of all students achievi 
mum percentages achieved on each criterion across 
that students in each country achieved on 


descriptions corresponding to a score of one and a score of two are 


criteria (Table 3.3). In most cases, the differen 


t creditab 


e levels of qua 


tion of the flow within and across 


described previously, data 


for scoring reliability. 


specific criteria. In general, 


ciency and information management. Criteria relating 
as text and image formatting and use 


cally included a hierarchy from little or no control at 
technical features to enhance the communicative impact of 
hus focused on ability to use the technical features for 
of skills. Criteria relating to 
information to suit audience 
tting information irrelevant to it), and 
for dichotomous scoring as 
thers allowed for partial credit scoring 
ts. 


s depended onthe nature of 


to support communication 
the 
the scoring criteria related 


eir 


ng 


t meant that the tasks could be accessed 
e design also allowed students across this 
ferent levels of achievement against the CIL scale, as evident in the 
iteria (Table 3.3). 


h occupy a single row because they are dichotomous criteria (scored 
ng to a score of one is | 
ia 1, 2, and 3 are partial-credit criteria (scored as zero, one, or two): 


ncluded for each of these 


included for each of these 
ity within the partial-credit 


criteria correspond to different proficiency levels on the CIL scale. For example, the description 


of a score of one on Criterion 1 is shown at Level 1 (4 
score of two on the same criterion is shown at Level 4 ( 


The lower category for each of two partial-credit scori 
to Level 1 on the CIL scale (Table 3.3). These both related to studen 


page elements and reflected students’ familiarity with 


139 scale poin 
736 scale poin 


ng criteria for t 


ts) and the description of a 


ts). 


he webpage corresponded 
ts’ control over the role of 


the basic conventions of using one of size, 


PREPARING FOR LIFE INA DIGITAL WORLD 


72 


‘sasayjuated ul seadde Sjoiia Ppsepueys :Sd}0N 


ased 9uyj JO (aunqeay 8ulpueig Ase||loue 
ue Ue Jayje4) JUIOd |ed04 e SAWIODEq 


1 EY} OS UOIJEDO| JO 9ZIS Sj! JEWWIOS UOIZEWIOJU! UOIELWIOJU! 
pue adedqam e Uo 089] e apn|du| asf—0807] BuqeatD BUIDNPOld Z'E (O°?) ZE (ZT) £8 ($0) €Z C/T 6ev T 
asedqam 
& JO} 9]}1} DY} Se 9jOI Sj] aJENIUNWWOD UOIEWIOJU! UOHELWIOJU! 
0} 4X9} UO!}ISO 40 eEUIO4 9sf\—aWeu pueg Bul}eo1D BUIDNPOd Ze (TZ) €€ (CT) Z8 (v0) L9 c/T L6v L 
a\qepea 
pue seajo Ajjesauas Ss! 3! Jey} OS (3uIqqeW40J/3NOAe]) UOHEWIOJU! UOHEWIOJU! 
adedqam e UO 3X9} 4no Ae] puke JeUIO4 AjIqepeay —jxa] BUIWIOJSUe | BuIDNpOld TE (ST) 8T (€T) 28 (SO) 19 Gp) OTS G 
asedqam e JO} 
9/41}. dU} Se ajOI S$} Aj4e9|9 9}9|UNWLUOD UONEWIOJU! UONEWIOJU! 
0} 4x9}. UO!JISOd pue yetWwJo4 9S/}—SWeU PUeg BuIyea1D BUIDNPOld Z'E (9T) ZT (9T) pL (S'0) 9S C/C ES G 
Aqligepea. 
yuoddns 0} a8edqam e uo syuawaja 
}SOW JO SJO}OD puNnoj8yDeq pue UOIEWIOJU! UONEWIOJU! 
7X9}. UsaMyaq }seJ}UOD JUa|DYJNs Ajddy 4SE1}UOD —} x0 8ul}ea1D BUIDNPOld Z'E (VT) 9T (ST) 8Z (SO) TS WAL CES C 
a|qepeas pue sea|d S141 yey} Os (3uIqqeW4OJ/3NOAe]) UOIEWIOJU! UOIEWIOJU! 
adedgam e UO 4x9} 3NO Ae] pue Jetuso+ Ayiqepeay —jxo| BUILIOJSUe | BUIDNPOld TE (OT) 6 (7) vZ (SO) OS Ge OVS G 
asedqam e uo a8eul asf —uoldisosap UO!JEWIOJUI UO!JEWIOJU! 
Ajdequdwa|duwoo e pue 1x9} apnjou| pue ojoud pueg SulW4OJsuel | sulDnpoig T's (ZT)6 (LZ) 99 (SO) Tv T/T 98S c 
asedqam quaWUsl|e UOIJEWIOJU! UOIJEUWIOJU! 
2 JO}. jNOAe| BAISAYOd e 9}34D /jnoke| adedqa\ BUIWOJSUE LL BUIDNPOd TE (CT) ST (OT) TL (SO) 8€ T/T 86S € 
asedqam e uo uoljed0] a|qeyns e 4xa4 UOIZEWIOJU! UOHEWIOJU! 
0} |JeWa Ue WY 4x9} payl9ads JaJsued | uoldlasap pueg BuIqea1D BUIDNPOd Z'€ (SO)Z (8T) SS (v0) Le T/T vv9 € 
Jea|d aqisqam au} 
JO aunqeay 8ulpuesig Aje}jioue ue se ajou 
S}1 DEW O} UOIJEDO] PUP 9ZIS S}! JEWOS UOHEWIOJU! UOHEWIOJU! 
pue adedqam e uo oso] e apn|du| aS—080] BUI}e31D BUIDNPOJd Z'E (OT)9 (VT) 6t (£0) £1 GG eye v 
uollay4 uolay4d 3994109 sjuapnys 
sulAaiyoe sulAaiyoe jo a8e}Uuads10d 3409S 
asejuadied | a8ejuadsed asesane uunuuixeu | AjjNoyyIp 
Joydisosaq UOLId}4D padsy pues WINWIUI | wnuIxel| 8T0Z STIDI /2409S ajeos TD | janeq 


4941109 JUaDIad [|DJAAO PUD SadUAsa{AJ YJOMALUD YIM DIJAIIO BULOIS ¥S0}-ABH| ajAwIoXF ‘EE alqvl 


STUDENTS’ COMPUTER AND INFORMATION LITERACY 


position, or formatting t 


of the webpa 


countries were able 
of the textbox to ind 


this, students de 


details of this 


webpages ex 
one score poi 


Level 2 achievement on 


of the forma 
(Criterion 3, 


ge. For 


Three other scoring criteria corresponded 
Text-Contrast, was dichotomous and appears at Leve 
51 percent of students were able to demonstrate some 
that most text e 


process (F 


hibited suft 
nt. 


tting tools 


to create a textbox con 
icate its role as the tit 
name-Use) was achieved by 56 percent of 
monstrated contro 
more clearly communicate its role as 


o denote the prominence of information. For Criterion 1, Logo-Use, 73 
percent of students on average across all countries could include the logo as a prominent feature 


over 
the tit 


ements in the webpage contrasted su 
aid readability. The ICILS scoring syste 
Contrast based on an adaptation of relevant criteria in the Web Contents Accessibi 
2.0 (WCAG 2.0; World Wide Web Consortium 2019). The ICI 


the scale was also exemp 
(e.g., text size and bolding 


Text-Readability). Students who cou 


readability for some elements received one score 


apply format 


61 percent and 50 percent of students achieved o 


criterion. 


At Level 3, s 


for purpose. 


Three 


dichotomous scoring criteria exempli 
students to demonstrate technical proficiency with an emphasis 
Criterion 7, Webpage layout/alignment, required students to i 


the textbox 


to Level 2 achi 


m automatically 


nsidered as complete produc 


by using both 


evement on the Cl 


2 only. 


ficiently with the 


generated a 


LS technica 


the score. S 
to be read c 


Criterion 2, Band name-Use, 67 percent of students on average across all 
taining the band name and demonstrate some control 
e for the webpage. Full credit on this criterion (Band 
students on average across all countries. To achieve 
position and formatting to 
e of the webpage representing Level 2 of the CIL Scale. 


L scale. One of these, 
On average across a 
planning in their use of color 
background color to 
suggested score for Text- 


countries 
and ensure 


lity Guidelines 


report provides full 
raillon et al. 2020). Human scorers reviewed the automatically generated 
suggested score for each webpage and could either accept or modify 
ficient color contrast for most text elements 


tudents whose 
early received 


ified by webpages with evide 
to support the readability of 
Id use the formatting tools to support text 
point while students who cou 
ting to all text elements received two score points. On average across all countries, 
ne and two score points respectively on this 


tudents’ execution of webpage design shows greater control and independent 
planning than at Levels 1 and 2. The control over webpage elements typi 
of independent planning extending beyond completion of the procedural aspects of the task. 
In essence, Level 3 webpages could be co 


specified elements on the webpage™*: the band competition logo, band description 
photo. In addition, students needed to demonstrate control of the overall flow of information 


by arranging and manipula 
all countries, 38 percent achieved full credi 


Cri 
text 


transcribed from the email to the page. Fu 
the band description text on the webpage 
included the signoff message in the email (“T 
as the si 


terion 5, Band descripti 
describing the ban 
3.8) on to the band webpag 


ting the elemen 


ts to create a 
t on this criteri 


on text, assessed the accuracy with 
d from 


14 See Figure 3.8 for the elements described in the emai 


, noting that use of the page 


on. 


which students 


exactly matched that in the email. 
hanks a lot!”) recei 


ved 


iterion. Criterion 


cally showed evidence 


ts that were largely fit 


nce of the use 
text elements 


d consistently 


fy Level 3 achievement. Each of these criteria required 
on information management. 
nclude at least two of three 


text, and band 


harmonious layout. On average across 


replicated the 


the email students were provided as part of the task (see Figure 
e. The text could be copied and pasted or (somewhat less efficiently) 
| credit was awarded on this criter 


ion only when 
Students who 


no credit for this criterion, 
gnoff message was deemed to be irrelevant to the webpage. On average across all 
countries, 27 percent of students achieved full credit on this cr 


6, Band photo 


itle was scored separately. 


PREPARING FOR LIFE INA DIGITAL WORLD 


and description-Use, assessed the degree to which students communicated a clear relationship 
between the band description text and the band photo on the webpage. This relationship was 
typically demonstrated by positioning the text and photo close to each other, relative to the 
other elements on the page. On average across all countries, 41 percent of students achieved 
full credit on this criterion. 


Full credit on Criterion 1, Logo-Use, is an example of Level 4, the highest level of achievement 
onthe ClL scale. Students achieving full credit (two score points) showed careful and deliberate 
use of position and size to make the role of the band competition logo an ancillary branding 
feature, rather than a prominent feature, of the webpage. Achievement at this level is evidence of 
students’ understanding the role of the webpage as a subpage of the band competition website 
and the importance of directing the viewer's attention to the most relevant information given 
the role of the webpage in the broader context of the website. On average across all countries, 
13 percent of students achieved full credit on this criterion. 


Comparison of CIL across countries 


Distribution of student achievement scores 


Across countries, the average student achievement scores on the CIL scale ranged from 395 to 
553 scale points, thereby forming a range that spanned a standard of proficiency Below Level 
1 to a standard of proficiency within Level 3. This range was equivalent to approximately 1.5 
standard deviations (Table 3.4). 


Differences in the within-country student score distributions tended to be larger in countries 
with lower average achievement than in countries with higher average achievement, and the 
variation in student CIL scores within countries was greater than that across countries (Table 
3.4). The distance between the lowest five percent and the highest five percent of CIL scores 
across countries ranged from 216 to 347 scale points (with a median of 269 scale points), in 
comparison to a range of average scores across all countries of 157 scale points. 


The differences between the average scores of adjacent countries were between two and 18 
scale points with the exception of a difference of 55 scale points between the average scores 
of students in Uruguay and Kazakhstan. 


CIL relative to the ICT development index for each country 


As additional context, we also calculated the average age of students in ICILS countries and then 
provide ICT development index (IDI) scores for each country® (Table 3.4). 


In ICILS 2013, we reported that that higher IDI scores were typically associated with higher 
CIL scores across countries (Fraillon et al. 2014). In ICILS 2018 the association between CIL 
achievement and the IDI scores across countries was again strong, with a Pearson’s correlation 
coefficient of 0.72. As in ICILS 2013 it is important to take into account the relatively small 
number of countries when interpreting these results. 


15 The IDI is a composite index that incorporates 11 different indicators relating to ICT readiness (infrastructure, access), 
ICT usage (individuals using the internet), and proxy indicators of ICT skills (adult literacy, secondary and tertiary 
enrollment). Each country is given a score out of 10 that can be used to provide a benchmarking measure with 
which to compare ICT development levels with other countries and within countries over time. Countries are ranked 
according to their IDI score. 
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Achievement across countries with respect to proficiency levels 


Across all countries, 80 percent of s 
1, 2, and 3 (Table 3.5). Overall, however, the distri 


and benchmarki 
is in Level 2 in al 


ng pa 
coun 


Although maj 
variation in th 
percentage of 
above Level 2 
Level 2 (i.e., at 
above Level 2 is 


or 


S 


ower 


ities O 
e distrib 
tuden 


Level 1 


rticipants sits 


f students in most countries 
ution of perce 


Trends in CIL achievement 


he ICILS 2018 test 
items. This meant that 
on the scale establish 
f 
in ICILS 2018. Three 
within each cycle to al 


The differences in ave 


we could report student C 
ed in 2013, and also com 


ow valid comparisons of s 


necessary sample par 


scale points or less) an 


d not statistically significa 


tudents achieved scores that 


ntages across countries. Int 
ts at Level 4 (Korea, Denmar 


bution of student sco 


argely within Level 2. The highest pe 
tries and benchmarking partici 


pants except 


ts below Level 2. 


rage CIL achievement scores in each of the three coun 
ticipation requirements in each of ICILS 2013 and 2018 were small (11 
nt (Table 3.6). However, in Chi 


for Uruguay and Kazakh 


had CIL scores in Level 2, there was 
hree countries with the h 
, and Finland) the proportion of stu 
(i.e., at Levels 3 and 4 combined) is higher than the proportion of students 
or Below Level 1). Across all other countries, the proportion of stu 
than the proportion of studen 


placed them within CIL Levels 


res across countries 
rcentage of students 
stan. 


some 
ighest 
dents 
below 
dents 


included three secure CIL test modules from ICILS 2013 comprising 61 
Lachievement scores for the current ICILS cycle 
pare changes in CIL achievement across these 
rst two cycles of ICILS. Four of the countries that participated in ICILS 2013 also participated 
of these countries met the necessary sample participa 


tion requirements 


tudents’ CIL achievement across the two cycles. 


tries that met the 


le, the percentage 


of students achieving at Level 2 or above decreased by seven percentage points between 2013 


and 2018 and this dif 


ference was statistically significant. The difference in 


the percentage of 


students achieving at Level 2 or above in Germany and Korea did not change significantly 
between 2013 and 2018 (Table 3.6). 


Variation in CIL across countries with respect to student background 


characteristics 


Int 
soci 


his sectio 


Our focus at 
achievement as report 
associa 
associa 
bac 


ted wi 


school-level factors.) 


Gender and CIL 


Previous su 
The Australi 
of year 6 an 
year 10 male studen 
Assessment of Educ 
reported higher ach 


was reported that “the 
higher than those o 


al backgrounds (such as gender and socioeco 


this point is on student characteris 


th students’ socioeconomic status ( 
ground, and the language students spoke at 
based on regression analyses, of the relationshi 


rveys of digital | 


dyear 10 fema 
tsine 
ation 


edin large-scale assessme 


tion, National Center for 
average ClL scale scores of fe 


iteracies have reported that female students outperforn 
an triennial sample assessments of ICT literacy reported that the avera 
e students was statistically significantly higher than th 
ach of 2008, 2011, 2014, and 2017 (ACARA 2018). The US National 
Progress sample assessment of Technology and 
ievement scores for female grade 8 students in ICT in both 2014 
(US Department of Educa 


Education Statistics 20 
male students were sta 


n we address Research Question CIL 4: What aspects of students’ personal and 
nomic background) are related to students’ CIL? 


tics that are commonly associated with student 
nts suchas ICILS. In this section we report onthe 
tions between students’ CIL and student gender, and between students CIL and variables 
SES), whether or not students had an immigrant 
home. (See Chapter 7 for afurther investigation, 
ps between student CIL and student-level and 


e students. 
ge achievement 
at of year 6 and 


nma 


Engineering Literacy 
Land 2018 
LS 2013 it 


ignificantly 


18). In IC 
tistically s 


f male students in all countries except Turkey and Thailand” (Fraillon et al. 
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2014, p. 102) and that in those two countries the difference in performance between male and 
female students was not statistically significant. 


InICILS 2018, the average ClL scale scores of female students were statistically significantly higher 
than those of male students in all countries and benchmarking participants except Chile, Uruguay, 
and North Rhine-Westphalia (Germany). In these three participants, there was no statistically 
significant difference between the average scores of female students and male students (Table 3.7). 
On average across all countries, the average score for female students was 505 scale points and 
for male students it was 488 scale points, an average difference of 18 scale points and equivalent 
to about one fifth of the ICILS standard deviation. 


The magnitude of the statistically significant differences in achievement between female and 
male students within countries and benchmarking participants ranged from six scale points in 


Moscow (Russ 


ian Federation) to 39 scale points in Korea.*¢ 


Home background indicators and CIL 


Socioeconomic background 


Socioeconomic background is a construct regarded as manifest in 
wealth (Hauser 1994). While it is widely regarded internationa 


range of learni 


Hauser 1994 
1985; Mueller 
caveats relatin 


occupation, education, and 
ly as an important correlate of a 


ng outcomes (Sirin 2005; Woessmann 2004), there is no scholarly consensus on 
which measures should be used for capturing family background (Entwistle and Astone 1994; 
and no agreed standards for creating composite measures of SES (Gottfried 
and Parcel 1981). Furthermore, in the context of international studies, there are 
g to the validity and cross-national comparability of socioeconomic background 
measures (Buchmann 2002). In this chapter, our consideration of the influence of socioeconomic 


background o 
achievement. 


In ICILS 2013 


higher CIL proficie 


and report on 
student questi 
the number of 


used in ICILS 2013. 


n CIL focuses on within-country associations between indicators of SES and 


, ‘[c]haracteristics reflecting higher socioeconomic status were associated with 
ncy both within and across countries” (Fraillon et al. 2014, p. 250). To measure 


socioeconomic background during ICILS 2018, we used responses from the 
onnaire. These related to parental occupational status, parental education, and 


books in the home, and were the same three socioecono 


The ICILS student questionnaire collected data on parental occupationa 


that allowed students to give open-ended responses. T 


according to t 
(International 


mic background variables 


status through questions 


he students’ responses were classified 


he International Standard Classification of Occupations (ISCO-08) framework 


Labour Organization 2007). Research indicates rela 


tively high consistencies 


between data on parental occupation collected from students and from parents (Schulz 2006; 


Vereecken and Vandegehuchte 2003). 


To generate a continuous measure of occupational status, Ganzeboom et al. (1992) coded the 
order to derive their international socioeconomic index (SEI). The SEI provides 
a cross-nationally comparable framework for organizing occupations in a hierarchical order 
according to their occupational status. We assigned SEI scores to each parent’s occupation and 


ISCO codes in 


then, for each 


student with two parents, took the higher of the two S 


El scores as the indicator 


score. For students from single-parent families, the one score served as the indicator. 


16 The nonsignificant differences were in Chile (eight scale points), Uruguay (five scale points), and North Rhine- 
Westphalia (Germany) (four scale points). 
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The S 


occupation results in terms of broader categories, we divided the S 
based on international cut-off points. These were “low-medium occupational status” (below 50 


score points) and “med 


To measure the educa 


used predefined categori 
ted with reference to the Intern 


construc 
and consisted of “ISCED 
ISCED 2” (OECD 1999; 
we used the highest ISC 
summarizing the associa 
achi 
or below 


"and “ISC 


As a measure of ho 
home. 
“26 to 
relatio 
two Ca 


We fo 


tegories: “below 2 


horizo 


es denoting ed 


6, 7, or 87 “ISC ED 


ED 4 or 5) “ISCED 3? “ISC 
UNESCO 2006). When students provided 


ED level as the indicator of parental educa 


evement, we used two categories of parental education: “below 
ED 6, 
me literacy resources, we used stu 

umber of books was broken down into six categories: “O to 10 books, “11 to 25 books” 
100 books,” “101 to 200 books,’ and “mo 
nship between the number of books in the h 


und statistically significant associations 
background variables and CIL across all countries 
ntal graphs in these types of tables indicate 


tion between the highest level of parental 


» 


7, or 8 (Bachelor’s degree or higher) 


dents’ repor 


ret 


6 books” and “26 books and above.” 


(Ta 
the 


El scale is continuous and ranges from 16 to 90 score points. To describe the parental 


data for both their parents, 
tional attainment, and when 
education and students’ CIL 
SCE 


ts of 


han 200 books.’ 
ome and students’ Cl 


El scale into two groups 


ium-high occupational status” (50 score points and above). 


tional attainment of each parent (based on the student responses), we 
ucational levels in each country. These categories were 
ational Standard Classification of Education (ISCE 


D) 
2; and “did not complete 


Wn 


D 6 (short-cycle tertiary 


number of books in the 


When summarizing the 
L achievement, we used 


between each of the three socioeconomic 
ble 3.8). (As a brief explanatory note, the 
magnitude [in CIL scale points], direction, 


and statistical significance of the differences between the average scores of students in each 
group. For each of the variables, green bars indicate a statistically significant difference in student 


CIL in favor of one grou 
another group.) 


p, and red bars indicate a sta 


tistically significant difference in favor of 


For each of the three socioeconomic background variables in each country, and overall across 


countries, the average Cl 


higher than that of students in the 
tween groups for all three variab 


be 
On 
paren 
poi 
The 
terti 


6 
a 


average across all countries, t 


Lscores of 


es varied across countries. 


Luxembourg. 


students in the “higher” groups were statistically significantly 
“lower” groups. However, the magnitude of the differences 


he difference between students in the highest and lowest 
tal occupation categories was 36 CIL scale points, with differences ranging from 18 scale 
nts in Korea to 51 scale points in 


ifference between the average CIL scale scores of students in the lower (short-cycle 
ry or below) and in the higher (Bachelor’s degree or higher) parental education groups on 


average across all countries was 31 scale points, with the minimum difference of 15 scale points 


in Fi 


Cro 
hav 


nland andt 


ss-nationall 


home on average was 50 scale poin 
of 63 scale points i 


and amaximum 


All three indicators of students’ S 


included in the multileve 


y, the difference be 


he maximum of 47 scale points in Chile and Uruguay. 


ts, with a minimum difference of 3 
Luxembourg. 


n 


| regression analyses presented in Chapter 7). 


ES contributed to a composite index of S 


tween the average CIL scale scores of students who reported 
ing 26 or more books at home and those students who reported fewer than 26 books at 


1 scale points in Portugal 


ES (this index is 
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Computers at home 


The ICILS 2018 assessment framework explains that the CIL construct assessed in IC 
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CHAPTER 4: 


Students’ computational thinking 


Chapter highlights 


Computational thinking (CT) achievement can be described as increasing according to the 
following progression: 


e At the lower region of the scale, students demonstrate a functional working knowledge of 
computation as input and output. They record data from observed outputs and implement 
mplete solutions to simple coding problems. 
the middle region of the scale, students demonstrate an understanding of computation 
as enabling practical solutions to real-world problems. They systematically associate inputs 
with outputs when planning solutions, and implement complete solutions to complex 
coding using non-linear logic. 


e Attheupper region of the scale, students demonstrate an understanding of computation as 
ageneralizable problem-solving framework. They infer the relationship between observed 
inputs and outputs to evaluate solutions. They implement elegant and efficient solutions 
to complex coding problems using repeat and conditional statements. 


Eight countries and one benchmarking participant completed the ICILS CT option. Students’ 
CT varied more within countries than across countries. 


e The range between the lowest five percent and the highest 95 percent of students’ CT 
scores within countries varied between 266 scale points (in Portugal) and 371 scale points 
(in Korea). (Table 4.1) 


e The difference between the highest and lowest average CT scores across countries was 
76 scale points. (Table 4.1) 


CT achievement tended to be higher among male students. 


e Across all countries the average CT scale scores of male students was statistically 
significantly higher than that of female students. (Table 4.2) 


e However, statistically significant differences in the average CT scale scores between 
female and male students were found in only two countries. In one of those countries 
the difference was in favor of female students and in the other it was in favor of male 
students. (Table 4.2) 


Socioeconomic status (SES), denoted by parental occupation, parental education, and number 
of books in the home, was significantly positively associated with student CT. 


e In all countries, students in the high SES groups scored significantly higher on the CT 
scale than those in the lower SES groups. (Table 4.3) 


Immigrant background and language background were associated with student CT 
achievement. 


e Insixof seven countries, students from non-immigrant families had statistically significantly 
higher CT scale scores than students from immigrant families. (Table 4.4) 


e In five of seven countries, students who reported mainly speaking the language of the 


CILS test at home had statistically significantly higher CT scale scores than those who 
reported speaking another language at home. (Table 4.4) 
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Access to computers at home and years’ experience using computers were associated with 


student 


na 
stati 
two 
na 
com 
havi 


Cl. 


| countries, students who reported having two or more computers at home had 
stically significantly higher CT scores than students who reported having fewer than 
computers at home. (Table 4.5) 

| countries, students who reported having five years or more experience using 
puters had statistically significantly higher CT scale scores than those who reported 
ng less than five years’ experience. (Table 4.5) 


Student CT achievement was strongly associated with student computer and information 
literacy (CIL) achievement. 


e¢ Onaverage across all countries, the correlation between students’ ClL and CT scale scores 


Was 


0.82. (Table 4.6) 


e Thecorrelation between students’ CIL and CT scale scores varied between 0.74 and 0.89 
across countries. (Table 4.6) 
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of the CT scale should not be directly compared to the levels in the CIL scale, as they have been 
developed using a different process and the scale metrics are not comparable. 


The lower region of the CT scale was defined as the region below 459 scale points, the middle 


region is that between 459 and 589 scale points (i 
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can, for example: 
rid problem solving; 
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their solutions. The stimulus area comprised three separate parts: the code blocks space (at 
the bottom left of the screen), the farm drone display space (the 9 x 9 grid at the top left), and 
the work space (the central space where code blocks could be arranged to form an algorithm). 


All tasks in the farm drone module presented students with the same interface design, with 
variations in the configuration of the farm, the task objectives, the available code block functions, 
and the state of the work space. The work space was presented as empty (with only the fixed 
“when run” command present) for tasks that required students to create code sequences. The 
work space was presented with pre-populated algorithms for tasks that required students to 
debug code. 


Students could drag code blocks into the work space. Code blocks connected to the “when 
run” code block would send instructions to the drone when the green “run program” button 
was clicked. Students could reset the state of the drone and the farm by clicking the blue reset 
button. They could also reset the state of the work space by clicking the orange reset button. 


The complexity of the tasks increased progressively through the farm drone module. The 
complexity of each task is influenced by the following set of key characteristics: 

e The task type (code creation or debugging); 

e The variety of available code functions (movement, action, repeat, conditional): 


e The number of targets (a target is a tile requiring a specific action to be completed over it 
uch as dropping water, seed, or fertilizer); 


n 


e The number of different target types (dirt, low, or high crops); 
e Whether or not any given target required more than one action to be completed over it; 


e The layout configuration of the targets (single or multiple rows); and 


e The number of different materials to be dropped on targets (water, seed, fertilizer). 


Figure 4.1: Example CT Task 1 with framework references and overall percent correct 
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Figure 4.1: Example CT Task 1 with framework references and overall percent correct (contd.) 


Score CT scale CT scale ICILS 2018 average 
region difficulty percentage correct responses 
At least one of three points Lower 353 86 (0.3) 
At least two of three points Lower 396 77 (0.4) 
Three points Upper 613 27 (0.5) 
ICILS assessment framework reference 
22 Operationalizing solutions 
Developing algorithms, programs, and interfaces 


Country Percentage scoring | Percentagescoring | Percentage scoring 
one out of two out of three points 
three points three points 
Denmark? * 92 (0.5) 83 (0.9 26 (1.2) 
Finland 87 (1.0) 80 (1.2 29 (1.0) 
France 87 (0.8) 77 (1.0 40 (1.3) 
Germany 83 (1.2) 73 (1.2 18 (1.2) 
orea, Republic of 90 (0.8) 86 (1.0 39 (1.9) 
Luxembourg 76 (0.5) 66 (0.5 16 (0.3) 
Portugal’! ? 88 (0.8) 78 (4.4 20 (1.1) 
Not meeting sample participation requirements 
United States 86 (0.6) 77 (0.7) 34 (1.1) 
Benchmarking participant meeting sample participation requirements 
orth Rhine-Westphalia (Germany) 84 (0.9) 73 (1.3) 17 (1.1) 


Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals 


may appear inconsistent. 
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y after replacement schools were included. 


it Nearly met guidelines for sampling participation rates after replacement schools were included. 
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from 76 percent to 92 percent. Achievement of a score of one on 
of the lower region on the CT scale. 
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Students who completed all of the objectives and used the repea 
code blocks more than the minimum necessary were awarded two score points; on average across 


f the lower region of the CT scale. 
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a ieved at least this. The percentages across countries 
and benchmarking participants varied from 73 percent to 86 percent. Achievement of a score 
of two on this task was in the upper end o 
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Students who could complete all of the task objectives using the repeat statement and with the 
fewest number of code blocks necessary were awarded the maximum of three score points; on 
average across all countries, 27 percent of students received the maximum score in this task. 
The percentages across countries and the benchmarking participant varied from 16 percent to 
AO percent. Achievement of a score of three on this task was in the upper region of the CT scale. 


Example CT Task 2 (Figure 4.2) is a high-complexity debugging task. The work space was pre- 
populated with a five-statement algorithm that students had to modify to complete the task 
objectives. 


Figure 4.2: Example CT Task 2 with framework references and overall percent correct 
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ICILS assessment framework reference 


2.1 Operationalizing solutions 


Planning and evaluating solutions 


98 


PREPARING FOR LIFE INA DIGITAL WORLD 


Figure 4.2: Example CT Task 2 with framework references and overall percent correct (contd.) 


Country Percentage scoring | Percentagescoring | Percentage scoring 
one out of two out of three points 
three points three points 
Denmark’ * 70 (1.3) 39 (1.2) 8 (0.7) 
Finland 66 (1.3) 44 (1.3) 9 (0.7) 
France 65 (1.2) 41 (1.1) 12 (0.7) 
Germany 60 (1.4) B2 (1:2) 7 (0.6) 
orea, Republic of 74 (1.4) 48 (1.6) 12 (0.8) 
Luxembourg 48 (0.6) 25 (0.4) 5 (0.3) 
Portugal'! * 61 (1.6) 31 (1.8) 3 (0A) 
Not meeting sample participation requirements 
United States 60 (1.1) 37 (0.9) 8 (0.4) 
Benchmarking participant meeting sample participation requirements 
orth Rhine-Westphalia (Germany) 61 (1.3) 32 (1.3) 6 (0.6) 
Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals 


may appear inconsistent. 


Met guidelines for sampling participation rates only after replacement schools were included. 
tT Nearly met guidelines for sampling participation rates after replacement schools were included. 
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Automated bus tasks (Example Tasks 3 and 4) 
Example CT Task 3 (Figure 4.3) represents the conceptualizing problems strand of the CT 
construct. The task presented students with a stopping distance simulator and the objective of 
finding the minimum viable braking distance under given conditions. Students were required 
to configure the flowchart according to the instructions to apply a set of conditions for the 
simulation. They then had to configure the braking distance and run the simulation to identify 


whether the bus stopped before, or crashed into, the rocks. 


Figure 4.3: Example CT Task 3 with framework references and overall percent correct 
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region difficulty percentage correct responses 

At least one of two points Middle 477 58 (0.4) 
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Figure 4.3: Example CT Task 3 with framework references and overall percent correct (contd.) 


Country Percentage scoring Percentage scoring 
one out of two points two points 
Denmark * 64 (1.3) 40 (1.3) 
Finland 62 (1.3) 37 3) 
France 48 (1.1) 27 (1.1) 
Germany 56 (1.2) 32 1) 
orea, Republic of 72 (1.2) 58 .2) 
Luxembourg 50 (0.6) 28 (0.5) 
Portugal! * 56 (1.4) 28 (1.4) 
Not meeting sample participation requirements 
United States 57 (1.0) 34 (0.9) 
Benchmarking participant meeting sample participation requirements 
orth Rhine-Westphalia (Germany) 55 (1.2) 29 (1.3) 
Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals 


may appear inconsistent. 


+ Met guidelines for sampling participation rates only after replacement schools were included. 
tt Nearly met guidelines for sampling participation rates after replacement schools were included. 
+ National defined population covers 90% to 95% of the national target population. 


Students could receive a score of zero, one, or two points on this task. One poin 
students who configured the simulator using conditions that were not specifie 
braking distance that was consistent with the configuration they used. Student 


ascore of two. 


the simulator correctly to the conditions and identified t 
two score points. Scores of one and two on this task were both indicative of achievement in the 
middle region of the CT scale, with a score of one near the lower end o 
of two at the upper end of the region. On average across all countries, 58 percent of students 
t least one on this task and 36 percent of students achieved a score of 
two. These percentages varied across countries and the benchmarking participant between 50 
nt for a score of at least one and between 27 percent and 58 percent for 


achieved a score of a 


percent and 72 perce 


nthe CT test, Examp 
Example CT Task 3 ab 


t was awarded to 
d but identified a 
s who configured 


he correct braking distance were awarded 


the simulator and req 
complete the decision tree with the labels provided. Students were required 
the labels from the left of the screen into the decision tree in a way that was consistent with 


uired students have some understanding of flowchartin 


f the region and a score 


e CT Task 4 (Figure 4.4) was presented to students as the task preceding 
ove. Example CT Task 4 was a planning task in preparation for configuring 


g conventions to 
to drag and drop 


both the logical sequence of the decision-making and the syntax of the decision tree. 
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Figure 4.4: Example CT Task 4 with framework references and overall percent correct 


Time 
Remaining 
00 mins 
Labels 
Reduce speed 20% 


Begin safety check 
Continue current speed 
bia ————s) ———s!| 


: 


End safety check 


The bus computer uses a safety check so that the bus does not crash into other vehicles. 
Drag and drop the labels onto the decision tree to show how the safety check should work. 


Click on > when you are ready to continue. 


Score CT scale CT scale ICILS 2018 average 
region difficulty percentage correct responses 

At least one of two points Middle 488 56 (0.4) 

Two points Upper 591 28 (0.4) 


ICILS assessment framework reference 


1.2 Conceptualizing problems 


Formulating and analyzing problems 
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Figure 4.4: Example CT Task 4 with framework references and overall percent correct (contd.) 


Country Percentage scoring Percentage scoring 
one out of two points two points 
Denmark? * 55 (1.8) 29 (1.0) 
Finland 56 (1.2) 27 (1.0 
France 55 (1.4) 30 (1.4) 
Germany 57 (14) 30 (1.0 
orea, Republic of 65 (1.2) 37 (1.3 
Luxembourg 51 (06) 24 (0.6 
Portugal't ? 50 (1.2) 20 (1.0 
Not meeting sample participation requirements 
United States 52 (0.7) 24 (0.6 
Benchmarking participant meeting sample participation requirements 
orth Rhine-Westphalia (Germany) 55 (1.1) 29 (1.0 


Notes: Standard errors appear in parentheses. Because r 


may appear inconsistent. 
+ Met guidelines for sampling participation rates on 
it Nearly met guidelines 


esults are rounded to 


y after replacement scho' 
or sampling participation rates after replacement sc 


1 National defined population covers 90% to 95% of the national target popu 


but placed the 


and “ 


Students could receive a score of zero, on 
the syntax correctly by dragging the “Yes” and “No” labels to th 
“Reduce speed 20%” and “Cont 


correct logic received two score points. This 


of the CT scale. On average across a 
least one on this task and 28 percen 
varied across countries and the benc 
for ascore of at least one and betwee 


e, or two points on 


inue current speed” labels under the wrong decision 
received one score point. Similarly, students who used the syntax incorrectly by placing the “Yes” 
0’ labels in the spaces below the decision point but placed the “Reduce speed 20%” and 
“Continue current speed” labels under the correc 
summary, students who could complete the decision tree with either the correct syntax or the 
correct logic received one score point. This was indicative of achievement in the middle region 
of the CT scale. Students who could complete the d 


was indicative of 


Comparison of CT across countries 


Distribution of student achievement scores 


When considering the distribution of studen 


countries and benchmarking participants, it is 
of countries participated in the ICILS CT international option (Table 4.1; note that the length of 


the bars in these tables reflect the spread of 
the within-country student sco 


differences in 


important to bea 


student scores 
re distributions). 


he nearest whole number, some totals 
ols were included. 


ools were included. 
ation. 


this task. Students who could use 


t decision also received one score point. In 


ecision tree with the correct syntax and the 


e spaces above the decision point 


achievement in the upper region 


| countries, 56 percent of students achieved a score of at 
t of students achieved a score of two. These percentages 
hmarking participant between 50 percent and 65 percent 
n 20 percent and 37 percent for a score of two. 


t achievement on the CT test for all participating 


rin mind that only asmall number 


within each country, highlighting 


The average country scores on the CIL scale ranged between 460 and 536 scale points; the 


range was equ 


the distribution of CIL, for CT the spread of scores wit 


ivalent to approximately 0.7 in 


ternational standard deviations (Table 4.1). Unlike 
hin countries does not appear to be clearly 


associated with achievement across countries (Table 4.1). However, as was observed for student 
on in student CT scores within countries was greater than that across countries. 
Across countries, the distance between the lowest five percent and the highest five percent of 


CIL, the variati 
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CT scores ranges between 266 and 371 scale points (with a median of 320 scale points or 3.2 
standard deviations), in comparison to a range of average scores across all countries of 76 scale 
points (equivalent to three quarters of an international standard deviation). 


CT relative to the ICT development index 


n Chapter 3 we reported that, on average, students had higher levels of CIL in countries with 
higher ICT development index (IDI) scores. The association between CT achievement and the 
D| scores across countries was moderate, with a Pearson's correlation coefficient of 0.43. 
However, the association between student CIL and country IDI in the same countries (i.e., only 
those countries in which students completed the tests of both CIL and CT) was weak, with a 
Pearson’s correlation coefficient of 0.23. A comparison of these correlation coefficients suggest 
that, across the countries taking part in the ICILS CT assessment, the association between the 
DI and a broader notion of digital competence (CIL) was lower than one might have predicted. 
This is potentially due to this smaller set of countries being relatively more homogeneous with 
respect to IDI and achievement than the broader set of ICILS 2018 participating countries. 


Variation in CT across countries with respect to student background 
characteristics 


In this section we address Research Question CT 4: What aspects of students’ personal and social 
backgrounds (such as gender and socioeconomic background) are related to students’ CT? 


Our focus is therefore on the associations between students’ CT and student gender, variables 
associated with students’ socioeconomic status, whether or not students had an immigrant 
background, and the language students spoke at home. Chapter 7 documents further 
investigation, based on regression modeling, of the relationships between student CT and 
student-level and school-level factors. 


Gender and CT 


In Chapter 3 we reported that the ClL scale scores of female students were statistically significantly 
higher than those of male students on average across all countries and within all countries and 
benchmarking participants except Chile, Uruguay, and North Rhine-Westphalia (Germany). 
A different pattern was evident for the relationship between gender and CT achievement to 
that of gender and CIL achievement. Across all countries the average CT scale score of male 
students was statistically significantly higher than that of female students (Table 4.2). However, 
this difference was not consistent at the country level. In Portugal and North Rhine-Westphalia 
(Germany) the average achievement of male students was statistically significantly higher than 

y 

e 

e 


that of female students and in Finland the CT achievement of female students was statistical 
significantly higher than that of male students. In France, Germany, Korea, and Luxembourg th 
average scores of male students appeared to be higher than those of female students but th 
differences were not statistically significant. In Chapter 8 we discuss the implications of the 
differences in achievement by gender found in both CIL and CT in further detail. 


Socioeconomic background 


In Chapter 3, details of how background data variables were collected and derived were presented. 
We reported statistically significant associations between each of the three socioeconomic 
background variables (parental occupation, parental education, and number of books in the 
home) and CIL across all countries. 


We found a similar pattern of association between socioeconomic background and CT 
achievement across all countries (Table 4.3) to that reported for CIL in Chapter 3. 
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For each of the three socioeconomic background variables in each country, and overall across 
countries and the benchmarking participant, the average CT scores of students in the “higher” 
groups were statistically significantly higher than those of students in the “lower” groups. However, 
the magnitude of the differences between groups for all three variables varied across countries. 


On average across all countries, the difference between students in the higher and lower parental 
occupation categories was 42 CT scale points, with the minimum difference being 21 scale points 
in Korea and the maximum difference being 67 scale points in Luxembourg. 


On average across all countries, the difference between the CT scores of students in the low 
short-cycle tertiary or below) and in the high (Bachelor’s degree or higher) parental education 
groups was 31 scale points, with the minimum difference being 19 scale points in Korea and 
the maximum difference being 46 scale points in Luxembourg. 


Cross-nationally, the difference between the average CT scale scores of students who reported 
having 26 or more books at home and those students who reported fewer than 26 books at 
home was 57 scale points, with the minimum difference being 40 scale points in Portugal and 
the maximum difference being 77 scale points in Germany. 


All three indicators of students’ socioeconomic status contributed to a composite index of 
socioeconomic status. This index is used in the multilevel regression analyses presented in 
Chapter 7. 


Immigrant status and language use 


n Chapter 3 we reported that the CIL scores of students without immigrant background tended 
to be higher than those with an immigrant background. We found similar results with CT where, 
for six of the seven countries that met the ICILS technical requirements, the students from 
non-immigrant family backgrounds had statistically significantly higher average CT scores than 
students from immigrant backgrounds (Table 4.4). On average across countries, the difference 
between students with immigrant backgrounds and those without was 46 CT scale points, with 
the minimum difference being 14 scale points in Portugal and the maximum difference being 
56 in Finland. 


n most participating countries, the majority of students indicated speaking the test language 
at home. Across countries, CT scores tended to be higher among students speaking the test 
anguage at home; the average difference was 47 scale points. For five of the seven participating 
countries meeting the technical requirements, we recorded statistically significant differences 
between students speaking the test language and those speaking other languages at home. The 
statistically significant positive differences varied from 38 scale points in Finland to 64 in France. 


Computers at home and experience using computers 


In Chapter 3 we noted that students with more computers at home tended to have higher CIL 
scores. Asimilar pattern of association can be seen when considering students’ CT achievement 
(Table 4.5). On average across countries, the CT scores of students reporting having two or more 
computers at home were 29 scale points higher than those who reported having fewer than two 
computers at home. This difference varied from 18 points in Portugal to 43 points in Germany 
and was statistically significant in all countries. 
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Similarly, students’ years of experience using computers was positively associated with CT. On 
average across all countries, the CT scores of students with five or more years of experience 
using computers were 23 scale points higher than those with less than five years’ experience. This 
difference varied from eight scale points in France to 50 scale points in Korea and was statistically 
significant in all countries but not statistically significant in the benchmarking participant North 
Rhine-Westphalia (Germany). 


The association between CT and CIL 


In this section we address Research Question CT 5: What is the association between students’ 
CIL and CT? 


CIL has an operational emphasis on 
and manage informatio 


is assessed through students’ respo 


students’ abilities to use computer technologies to collect 
n and to produce and exchange information. In the ICILS test of CIL, this 


nses to a broad range of tasks that focus on (receptive and 


productive) inforn 
Its focus | 
to [real-world] problems so that the solutio 
et al. 2019, p. 27). 
operationalized solution to a real-world pro 


d CT areq 


ClL an 
their d 
do sha 
one th 
descri 


efi 
re 


CIL an 


in achiev 


domains would lead to a posit 


On average across all countries, the correlatio 
was 0.82 (Table 4.6). This strong correlation be 
countries and varied from 0.74 in Korea to 0.89 


CIL and 
performi 


the average CT scores of stu 
he difference in student CT scale scores between s 
tude 
students with CIL of Level 3 and Level 


across al 
levels of 


and Below Leve 


or above 
scores of 


nation literacy ina digitale 


sonthe 


nvironment. CT isanew assessment co 


ementation, and evaluation of “algori 


nstruct to ICILS. 
thmic solutions 


ere 
bed 
1.1 (knowi 
dC 
thinking (t 
it was expected tha 


planning, formulati 


The ICILS 


uite di 


fferent achievemen 


on, imp 
n 


CT assessme 


t const 


nitions and descripti 
some common 
fore draws on und 
by CIL Aspect 1.1 
ng abou 
| draws 
hrough t 


ement within stude 


CT scores across a 


countries, t 
achievement varied 
ta 
. Across Cou 


studen 


achi 
stud 
reported 
their rela 


evemen 
ents’ 


t than between t 


featu 


t and understanding digi 
on studen 
he evaluati 
t, while the differences b 


nd 60 CT scale points (be 
ntries there was a general tendency for the differe 
between t 
ted the cor 


ts in adjacent 


ons as well as the tas 
res. They both are, a 
erstandings of 
knowing abou 


t and u 
tal sys 


ts’ literacy ski 
on of information, da 


etween 


ta, 


nts, the comm 
ive association 


between 


n be 
tween 


| countries as well as t 


ng within each CIL proficiency level across countries (Tab 


dents increase as the C 


from 90 CT scale poin 
tween 


blem and a sui 


ructs. 
s used 
nd can 
how comput 
nders 
tems). 
Is (in reading and 


onalities | 


in Fin 


only 
ers C 
tandi 


ng co 


and solutions 


n the foundat 


s could be operationalized with acomputer” 
t combines tasks involving planning 
te of visual coding tas 


This is evident through 
to assess each domain 
be, completed on 
an be used to so 


mpu 


Furthermore, achieve 
responding to tasks) and critical 
to problems). In IC 
the two domains would result in dif 
ions of achieveme 


ClL and CT ac 


L levels of stu 


ts (between s 


he average CT 


tween students’ CIL and CT sca 
CIL and CT scores was consiste 
land. We report the correlations between 
scale scores for student 
le 4. 


den 


hievement. 


Fraillon 
for a computer- 


S. 


examination of 
. However, they 
computer. Each 
ve problems as 
and CT Aspect 
ment in each of 


ter use 


LS 2018 
ferences 
the 


nt of 


e scores 
nt across 


nn 


3). Across all countrie 
ts increase. On averag 
tudents in adjacent C 
nts with CIL of Level 


(a) 


of PTH 


nce in average CT sca 


he lower levels of 


CIL levels of achieveme 
he higher levels. We fur 


Cc 
ac 
tio 


L and CT by gen 
ross coun 
nshi 


tries. Th 
ps to student gender are further discu 


der and found that they 
e consequences of the s 


nt to be larger 
ther investiga 


varied little from 


trong correlati 


relation between 


on betwee 
ssed in Chapter 8. 


the overall correlations 
nClLand CT and 


111 


STUDENTS’ COMPUTATIONAL THINKING 


‘uolqe|ndod 3aBJe} |EUOIJEU BU} JO %GS 01 %OS S49A09 uolje|ndod pauyap jeuojeN =; 


‘papNjoul 34am sjooups JUawaIe|da Jaye sayzes UOed|dIqJed Buljdwies Joy SauIjapIN Jaw AjJeaN 4, 
‘PapnN|oul dam sjooyps JUsade|da Jaqye AJUO $9}eJ UO! 


edioqued 8uljduwies sojsauljapins yap) | 


‘sadesjane dno 440d 0} ||EWIS 00} SJUapPNys JO AGUINN 


*SOSoU 


juased ul seadde suosia puepueys :Sd}0N 


(vvt) Ov9 (09) ZS (O'€) OS (Ov) O€ (28) Ove (T0'0) 180 (AueWJa5) el/eydysa/\-oulyy YIJON 
sjuawasiInbad uoljedioiqed ajdwes suijaouw JUedIDIWWed 3ulysewyouag 
(86) €Z9 (97) 76S (O'%) STS (9%) 6¢r (Sv) vee (T0'0) + zs0 seye4s pour 
sjuawasinbas uoedioiied ajdues 8uja9W JON 
(LS) v9 (97) S8S (TT) STS (QT) 6€r (O€) 6re (00'0) zsgO aBeIOAL BLOT SIDI 
(O'ET) 009 (ES) €SS (v7) Lép (Ze) Ter (ZZ) SE (TOO) 8Z0 1 lesnyod 
(EST) TS9 (EE) SBS (67) OLS (T°?) O€P (O€) TEE (LOO) O80 sINOqUIaXN] 
(62) 859 (S'€) 26S (fv) SES (6S) €9r (68) OZE (TOO) vZ0 jo dijqnday ‘easo> 
(OZ) 659 (Cv) 82S (CE) ZOS (QE) OCP (OZ) 8¢€ (TOO) TSO AueWa 
v v (v'S) S09 (v7) €€S (CE) 8Sr (69) OLE (T00) Z80 eouel4 
(ST) 7S9 (LE) 16S (97) TS (67) O€ (WZ) 6ZE (TOO) 680 pue|ul4 
(SET) SO (SE) €8S (87) OTS (6%) Lev (QTL) O9€ (TOO) TSO 7 PEWS 
aaoge pue f |aA27] € |aAe7 Z [aA27 T [ae] T |aae7 
19 32 syuapnys 710 3e syuapnys 405 719 32 sjuapnys 119 7e sjuapnys ID Mojaq sjuapnys ID pue 
Jj 2409S | 5 asesaAY 3409S | asesaAV JO} 2409S 1D aseIaAY 4Oj 2409S | 5 aBeIaAY JOj 2409S 1D aBeIaAY 19 uolje|a4405 Asjyunoz 


SalJUNOD SSOJID [ana] AQualgYod J]D Yyova 1D squapN}s JOJ adUBULIOLJad | D a8vsaAD PUD T|D PUD [D UaaMjaq SUO!}D|AOD ‘9p a{qo 


112 PREPARING FOR LIFE INA DIGITAL WORLD 


References 


Fraillon, J., Ainley, J., Schulz, W., Duckworth, D., & Friedman, T. (2019). IEA International Computer and 
Information Literacy Study 2018 assessment framework. Cham, Switzerland: Springer. Retrieved from https:// 
www.springer.com/gp/book/9 783030193881. 


Fraillon, J., Schulz, W., Friedman, T., & Meyer, S. (Eds.). (2020). IEA International Computer and Information 
Literacy Study 2018 technical report. Amsterdam, The Netherlands: International Association for the 
Evaluation of Educational Achievement (IEA). Manuscript in preparation. 


TU. (2019). IDI 2017 rank [webpage]. Geneva, Switzerland: International Telecommunications Union. 
Retrieved from https://www.itu.int/net4/|TU-D/idi/2017/index.html. 


Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: 
ielsen & Lydiche. 


Von Davier, M., Gonzalez, E., & Mislevy, R. (2009). What are plausible values and why are they useful? 
nM. von Davier, & D. Hastedt (Eds.), ER! Monograph Series: Issues and Methodologies in Large-Scale 
Assessments, Volume 2 (pp. 9-36). Hamburg, Germany/Princeton, NJ: International Association for the 
Evaluation of Educational Achievement (!EA)/Educational Testing Service (ETS). Retrieved from http://www. 
ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_O2_Chapter_O1.pdf. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International 
License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution 
and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a 
link to the Creative Commons license and indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated 
otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use 
is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright 
holder. 


CHAPTER 5: 


Students’ engagement with information 
and communications technologies 


Chapter highlights 


Students were experienced users of information and communications technology (ICT). 


e Slightly fewer than half of the students in grade 8 had been using computers for five or 


more years. 


e Computer experience was associated with students’ computer and information literacy. 


(Table 5.1) 


Students frequently used ICT for general purposes. 


e Seven out of 10 grade 8 students used ICT on a daily basis outside school for general 


purposes. (Table 5.2) 


purposes but only one in five students reported ICT use on a daily basis for school-related 


e Student use of general applications in ICT was more frequent among those with five or 


e Most students used ICT at least 


downloaded music or watching videos. (Table 5.10) 


more years of computer experience, those currently studying a computer subject, and 
those with higher levels of computer literacy. (Table 5.5) 


once each week for leisure activities such as listening to 


e Approximately two thirds of students used ICT to access information about things of 
personal interest from the internet at least once each week. (Table 5.10) 


School-related use of ICT most ofte 


n involved internet searching and document production. 


e The most frequent school-related use of ICT was using the internet to do research. 


Approximately three students in 


five did this at least once per week. (Table 5.13) 


e About one quarter of the students used ICT on a weekly basis to collaborate with other 


students or organize their time a 


nd work. (Table 5.13) 


e¢ One quarter of the students used ICT on a weekly basis to prepare reports and essays. 


(Table 5.13) 


e The ICT tools that students most commonly used in a majority of lessons were computer- 
based information resources, word processing software, and presentation software. (Table 


5,17) 


Most students were confident users of ICT and saw benefits of ICT for society. 


e Four out of five students were 


confident about their ability to use ICT to search for 


information, insert an image into a document, and write or edit text for a school 


assignment. (Table 5.24) 


e There was little difference between male and female students in their confidence in using 


general ICT applications (Table 


5.28) but male students expressed greater confidence 


regarding their use of specialist ICT applications. (Table 5.29) 


e Confidence in using general ICT 


applications was associated with measured CIL and CT, 


but confidence in the use of specialist ICT applications was not. (Table 5.36) 
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e Most students tended to acknowledge positive outcomes of ICT for society, but around 
half of the students also agreed that ICT had some negative consequences for society. 
(Table 5.31) 

e Male students had greater expectations than female students of using ICT for work or 
study in the future. (Table 5.35) 
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Introduction 


The International Computer and Information Literacy Study (ICILS) 2018 investigated students’ 
experience of using information and communication technology (ICT), their frequency of using 
ICT for a range of different purposes at and outside of school, and their dispositions toward the 
use of ICT. This builds on the knowledge about variations in the extent and type of ICT use by 
students established in ICILS 2013. With large representative samples it is possible to report 
not only on levels and patterns of ICT engagement but on the relationships of ICT engagement 
with student attributes. 


3. 


Our examination of students’ engagement with ICT was informed by opportunity to learn, a 
construct that has featured in IEA large-scale international assessment studies over along period 
of time (Elliott and Bartlett 2016; Scheerens 2017; Schmidt et al. 2013). Opportunity to learn 
initially referred to the time allocated for students to be taught the concepts being assessed and 
the curriculum content that was the focus of that time. The construct evolved to take account 
of the enacted curriculum rather than the intended curriculum (Rowan and Correnti 2009) and 


whether students were actively engaged during that time (Fisher et al. 1981). 


MT] 


We based our investigation on the in-school and out-of-school time that students engaged with 
ICT because students learned about and developed skills in using ICT in both environments. Our 
focus was on the frequency with which students engaged in different types of activities rather 
than where that engagement took place. We distinguished between ICT engagement for general 
purposes and ICT engagement for school-related purposes. We also asked students about the 
content of the ICT learning they had experienced at school and aspects of their attitudes to ICT. 


Our concern was to examine the associations between students engagement with ICT and the 
computer and information literacy (CIL) and computational thinking (CT). This chapter inform 
Research Question 3: What are the relationships between students’ levels of access to, familiarit 
with, and self-reported proficiency in using computers and their CIL/CT? However, we are not sole 
interested in the relationships of these aspects with achievement in CIL/CT. Another purpose of 
CILS 2018 is to investigate the use of computers and other digital devices by students, as well 
as their attitudes toward the use of computer technologies. These frame the broader context 
in which computer technologies are used within and outside school. 


< DFT 


< 


Forms of engagement with ICT 


Following the taxonomy proposed by Fredericks et al. (2004), we use the term “engagement” to 
encompass behavioral engagement (i.e., how students use ICT and how often they use it) and 
emotional engagement (i.e., students’ attitudes toward and feelings about ICT). 


In order to assess behavioral engagement with ICT we investigated students’ general use 
of ICT and engagement with ICT for school-related purposes. Students’ general use of ICT 
encompassed overall frequency of use as well as use for three particular purposes: creating 
or editing information products, social communication and information exchange, and leisure 
a 
O 


ctivities. Student engagement with ICT for school-related purposes encompassed overall use 
f ICT for school-related purposes and patterns of ICT use for school-related purposes. Patterns 
of use for school-related purposes included the ICT tools that were used and the variations in 
ICT use across subject areas. 


Knowing about students’ experience of learning about ICT in school is an important aspect of 
discerning the enacted curriculum within educational systems. Some literature has argued that 
students are “digital natives” who learned to use ICT outside school (Prensky 2001). However, 
others have contended that there are important aspects of ICT use that are not familiar to 
students and need to be taught (Selwyn 2009). We asked students about the extent to which 
they had learned about particular aspects of CIL and CT at school. 


In order to assess emotional engagement, we investigated two m 
perceptions of ICT. The first aspect was students’ perceptions of themselves 
ICT self-efficacy. We asked students to indicate how well they felt th 
| tasks. Based on the results from ICILS 2013 (Fraillon et 
from these tasks. The first referred to ICT self-efficacy in relation to general applications 
(typically embodied in office applications) and the second referred to ICT 
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The scales that we used in analysis and reporting, based on the student questionnaire, were: 


e Students’ general engagement with ICT 


— Frequency of use of ge 


— Frequency of use of IC 
— Frequency of use of IC 


— Frequency of use of IC 


neral ICT applications 


— Frequency of use of specialist ICT applications 


T for social communication 


T for exchanging information 


T for accessing content from the internet 


e Student engagement with ICT for school-related purposes 


— Frequency of use of IC 


— Frequency of use of ge 


T for study purposes 


neral applications in class 


— Frequency of use of specialist applications in class 


e Extent of student learning about ICT at school 


— Extent to which students learned about CIL tasks at school 


— Extent to which students learned about CT tasks at school 


— ICT self-e 


ficacy regarding the use of general applications 


ficacy regarding the use of specialist applications 


— Attitudes to ICT futures 


e Perceptions of ICT 


— Perceptions of positive effects of ICT on society 


— Perceptions of negativ 


e effects of ICT on society 


— Perceptions of personal futures with ICT 


Student general engagement with ICT 


(ISCED 2) engaged in ICT- 


streaming or downloading 
information searching, and 
their “digital competences 


people in and outside school (Bul 
accompanied by a growing interest in how these technologies are 
he European Commission reported that 80 percent of students in 


The past four decades have seen substantial growth in the availability and use of ICT by young 
fin et al. 2016). Growth in student use of ICT has been 
being used (Bulfin et al. 2016). 


lower-secondary school 


based activities more frequently at home than at school (European 
Commission 2013). The report identified three groups of |CT-based ac 
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(European Commission 20138, p. 15). Scherer et al. (2017) identified 
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ted internet activities and 


CT for a wide range of activities. These profiles were associated 


ICILS 2013 has also been an important source for understanding and reporting students’ general 
ICT use. Bundsgaard and Gerick (2017) used latent class analysis of ICILS 2013 data to identify 
three clusters reflecting different types of students’ computer use. The 


largest cluster (72% of the 


sample) had average frequencies of school-related and recreational computer use. The next cluster 
(12%) had low frequencies of computer use for communication and study purposes. The third 
cluster (11%) had high frequencies of use in general and especially for exchanging information. 
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There was a significant association between computer experience and CIL in all participating 
countries (averaging seven points per year of experience) and asmaller but significant association 
between experience of using tablet devices and CIL in eight of the participating countries that 
met sample requirements (averaging two points per year of experience) (Table 5.1). There were 
significant associations between experience of using smartphones and CIL in 10 participating 
entities, but some were negative and some were positive so that on average there was no 
statistically significant effect. On average, student experience of using computers accounted 
for just five percent of the variance in CIL scores. 


Frequency of ICT use 


We computed the percentages of grade 8 students who reported using computers at least 
once a day in each of four categories: outside of school for school related purposes, outside of 
school for non-school related purposes, inside school for school related purposes, and inside 
school for non-school related purposes.’? Daily use of ICT for other (i.e., not school-related 
purposes outside school was the most frequent use in every country (Table 5.2). On average 70 
percent of grade 8 students reported daily use of ICT outside of school for other purposes. Of 
the participating educational systems the frequency was highest in Germany (83%) and North 
Rhine-Westphalia (Germany) (85%) and lowest in Kazakhstan (48%). The next most frequent 
category of daily ICT use was at school for other purposes, which was reported on average by 
29 percent of grade 8 students. Among the participating countries, daily use of ICT at schoo 
for other purposes was most frequent in Finland (56%) and Denmark (55%) and least frequent 
in Italy (4%), France (13%), and Germany (16%). 


Daily use of ICT for school-related purposes was less common than for other purposes. On 
average across participating countries, 18 percent of grade 8 students used ICT on a daily basis 
for school-related purposes at school and 21 percent of these students used ICT on a daily basis 
for school-related purposes outside of school (Table 5.2). Using ICT on a daily basis for school- 
related purposes at school was most frequently reported in Denmark (81%) and least frequently 
7% or less) in Germany, Korea, Portugal, and Italy. Daily use of ICT for school-related purposes 
outside of school was most frequent in Denmark (35%) and Moscow (Russian Federation) (40%) 
and least frequent in North Rhine-Westphalia (Germany) (9%), Portugal (10%), Korea (10%), and 
Germany (11%). 


It is also evident that in Denmark the frequency of daily use of ICT outside school for other 
purposes (79%) is similar to the frequency of daily use at school for school-related purposes 
81%) (Table 5.2). In contrast, there were large differences between the frequency of daily use 
of ICT outside school for other purposes and the frequency of daily use of ICT at school for 
school-related purposes in Germany (83% compared to 4%), Italy (77% compared to 7%), and 
France (76% compared to 8%). These differences possibly reflect the extent to which ICT is part 
of teaching and learning in school education and may provide an index of the emphasis on ICT 
in schooling at lower-secondary level. 


19 In ICILS 2013 we reported frequency of use on a weekly rather than a daily basis, and we did not separate usage for 
general purposes and school-related purposes. Therefore, the data for ICILS 2018 are not comparable with those 
reported for ICILS 2013, even for the three countries that met sampling requirements in both studies. 
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Table 5.2: Percentages of students reporting daily use of ICT in and outside school for school-related 


and other purposes 
Country Percentages of students who reported daily use of ICT: 
At school for At school for Outside of school Outside of school 
school-related other purposes for school-related for other 
purposes purposes purposes 
Chile 12 (0.9) V 27 (1.2) 4 (0.9) V 62 (15) V 
Denmark'? 81 (12) A 55 (14) A 35 (15) A 79 (1.0) A 
Finland 12 (10) V 56 (14) A 5 (09) V 79 (0.9) A 
France 8 (07) V 13 (11) VW 25 (0.9) A 76 (0.9) A 
Germany 4 (06) W 16 (1.2) W 1 (0.8) W 83 (0.9) A 
taly” 7 (06) VW 4 (05) VW 22 (0.9) 77 (1.0) A 
azakhstan* 24 (11) A 30 (1.1) 31 (12) A 48 (14) W 
orea, Republic of 5 (0.5) W 19 (10) V 0 (0.7) W 68 (1.0) V 
Luxembourg 18 (0.6) 33 (0.6) A 27 (0.5) A 66 (06) V 
Portugal’! * 7 (05) VW 36 (11) A 0 (0.7) VW 71 (1.3) 
Uruguay 15 (0.9) V 25 (14) V 33 (14) A 66 (1.6) V 
ICILS 2018 average 18 (0.2) Dom(O3) 21 (0.3) 70 (0.3) 
Not meeting sample participation requirements 
United States 43 (1.6) 28 (1.0) 29 (0.9) 66 (0.9) 
Benchmarking participants meeting sample participation requirements 
Moscow (Russian Federation) 22 (0.8) A 43 (11) A 40 (1.0) A 77 (1.3) A 
North Rhine-Westphalia (Germany) 3 (0.5) vV 19 (1.5) V 9 (08) W 85 (0.9) A 
Notes: Standard errors appear in parentheses. Because results are rounded National ICILS 2018 results are: 


to the nearest whole number, some t 


otals may appear inconsistent. 


Comparisons with ICILS 2018 only reported for countries or benchmarking 


participants meeting sample particip 


1 Met guidelines for sampling participation rates only after replacement 


schools were included. 


ation requirements. 


it Nearly met guidelines for sampling participation rates after replacement 


schools were included. 


t National defined population covers 90% to 95% of the national target 


population. 
? Country surveyed target grade in 


the first half of the school year. 


Use of ICT to create or edit information products 


A 


A 
V 
Vv 


More than 10 percentage points 
above average 


Significantly above average 
Significantly below average 
More than 10 percentage points 
below average 


On average across ICILS 2018 educational systems one third (33%) of grade 8 students used ICT 
to write or edit documents at least once each week, one fifth (21%) used |CT-based spreadsheets 
for calculations or graphing, and one fifth (19%) used ICT to develop slideshow presentations 


51%, and 38% respectively). 
nd North Rhine-Westphalia 


ied) 


Table 5.3). The prevalence of these uses of ICT 


T on aweekly basis was highest in Denmark (84%, 


These uses of ICT were less prevalent in Korea, Finland, Germany, 


(Germany). 


Based on the three comparable countries”? from ICILS 2013 (Fraillon et al. 2014, p. 133), it 
appears that there may have been a small increase in the weekly use of ICT to write or edit 
documents (notably in Germany), an increase in the weekly use of spreadsheets (again particularly 
Germany), and little change in the weekly use of ICT to develop slideshow presentations. 


20 The three countries were Chile, Germany, and Korea. Denmark participated in ICILS 2013, but did not satisfy 


sampling requirements. 
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On average across all countries, the specialist applications used at least weekly by the highest 
percentages of students were: recording or editing videos (28%), using drawing and painting 
software (20%), and producing or editing music (20%). Activities reported to be conducted on 
at least a weekly basis by smaller percentages of students were: writing computer programs or 

cripts (12%), and building or editing a webpage (8%). The prevalence of weekly use of music 
applications varied greatly from nearly two-fifths in Chile (88%) and Kazakhstan (39%) to less 
than one in 14 in Finland (4%), Denmark (7%), and Korea (7%). It is of interest that in Denmark, 
although there were high proportions of students reporting weekly use of general applications, 
there was only a low percentage of students indicating weekly use of ICT for music production 
r editing/building webpages. 


WY 


[e) 


Between 2013 and 2018 there appeared to have been increases in the weekly use of drawing, 
painting, and graphics software in Chile, Germany, and Korea. There were only small increases 
in the weekly writing of computer programs in these countries. 


The items were used to derive two IRT scales reflecting students’ use of general applications for 
activities and students’ use of specialist applications for activities, where higher scale scores reflected 
higher frequency of use. Both scales had satisfactory reliabilities with average Cronbach’s alpha 
coefficients across countries of 0.70 and 0.73, respectively (the item maps describing these 
scales are included in Figures F.2 and F.3 in Appendix F). 


We used these scale scores (set to metrics with a mean of 50 and a standard deviation of 10 for 
equally weighted participating countries) to investigate differences among countries in students’ 
use of general applications and specialist applications (Table 5.4). It was evident that the scale 
scores reflecting reported use of general applications were highest in Denmark and Kazakhstan 
and lowest in Korea, Finland, and North Rhine-Westphalia (Germany). We recorded the highest 
scale score of reported use of specialist applications in Kazakhstan and the lowest score in Finland. 


We also reviewed the associations between the scale scores representing the use of ICT 
applications and several aspects of students’ experience of, and expertise in, using computers. 
We compared, for each country, the mean scale scores for the frequency of using general ICT 
applications (including productivity software) for:?* 


e Students with less than five years of computer experience with those who had five or more 
years of computer experience; 


e Students who studied computer subjects (e.g., computing, computer science, information 
technology, informatics, or similar) in the current school year with those who did not study 
computer subjects; and 


e Students with CIL scores below Level 2 with students whose CIL scores were at Level 2 or 
above. 


21 The percentages of students in each subgroup are reported in Appendix E. 
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Use of ICT for social communication and exchange of information 
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The social communication items were: 
e Share news about current events on social media; 


e Communicate with friends, family, or other people using instant messaging, voice, or video 
chat (e.g., Skype, WhatsApp, or Viber); 


e Send texts or instant messages to friends, family, or other people; 


e Write posts and updates about what happens in your life on social media; 


e Post images or video in social networks or online communities (e.g., Facebook, Instagram, or 
YouTube): 


e Watch videos or images that other people have posted online; and 


e Send or forward information about events or activities to other people. 


The information exchange items were the following: 


e Ask questions on forums or Q&A (question and answer) websites; 


e Answer other people’s questions on forums or Q&A websites; and 


e Write posts for your own blog (e.g., using WordPress, Tumblr, or Blogger). 


The items were used to derive two IRT scales reflecting students’ use of ICT for social communication 
and students’ use of ICT for exchanging information, where higher scale scores reflected higher 
frequency of use. Both scales had satisfactory reliabilities with average Cronbach's alpha 
coefficients across countries of 0.77 and 0.75 respectively (the item maps describing these 
scales are included in Figures F.4 and F.5, Appendix F). 


An inspection of national mean scale scores for the frequency of social communication and for 
information exchange indicated small differences among countries on the social communication 
scale but larger differences among countries on the information exchange scale (Table 5.7). Scale 
scores reflecting the use of ICT for information exchange were relatively high in Kazakhstan and 
Chile and relatively low in Denmark and Finland. 


The average scale scores for the use of ICT for social communication were significantly higher 
for students who were experienced computer users than other students in all countries except 
Denmark, and the average difference between the two comparison groups was three scale 
points (Table 5.8). 


In only two countries (Kazakhstan and Portugal) did students who were currently studying 
computing subjects have higher scores on the ICT for social communication scale than those who 
were not studying ICT (Table 5.8). In Finland, students who were currently studying computer 
subjects had lower scores on the ICT for social communication scale than other students. In 
most countries there was no significant difference between the two groups of students. 


In four ICILS 2018 countries, students whose CIL scores were at Level 2 or above used ICT for 
social communication more frequently than students with CIL scores below Level 2. On average 
across educational systems the difference was just one scale point. The opposite was true for 
North Rhine-Westphalia (Germany). 


The scale score reflecting frequency of use of ICT for exchanging information was significantly 
higher for students who were experienced computer users than for other students in nine of 
the ICILS countries, but the average difference was only one scale point (Table 5.9). In just four 
of the countries, students who were currently studying computing subjects had higher scores 
on the use of ICT for exchanging information scale than those who were not. In 11 of the 
ICILS countries, students with CIL scores below Level 2 had higher scale scores for ICT use to 
exchange information than students whose CIL scores were at Level 2 or above. On average 
across countries the difference was three scale points. The largest differences were evident in 
Denmark (five points), Germany (four points), and Luxembourg (four points). 
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Use of ICT for leisure activities 


Prior research has shown that students tend to use ICT frequently for leisure activities (Tobias et 
al. 2011) and this was confirmed in ICILS 2013 (Fraillon et al. 2014, p. 143). ICILS 2018 included 
leisure activities that involved accessing content from the internet (but not necessarily for study 


or school) as well as recreationa 


| activities such as playing games, listening to downloaded music, 


or watching downloaded or streamed TV or movies. The ICILS 2018 student questionnaire 
asked students to indicate how often they used computers for leisure activities. For reporting 
purposes, we categorized these as the percentages who reported doing these activities at least 


once each week (Table 5.10). 


There was a high prevalence of using ICT for recreation on a weekly basis (Table 5.10). On 
average across ICILS 2018 countries, 83 percent of students used ICT to listen to downloaded 
or streamed music at least once each week, 71 percent used ICT to play single-player games at 
least once each week, and 68 percent used ICT to watch downloaded or streamed TV shows or 
movies on a weekly basis. Using ICT to watch downloaded or streamed TV shows or movies on 
a weekly basis was most common in Denmark (81%) and Moscow (Russian Federation) (83%), 


and least common in Korea (57%). 


Inthe three comparable countri 
to have been an increase in the 


es from ICILS 2013 (Chile, Germany, and Korea) there appeared 
use of ICT to access “the internet to find out about places to go 


or activities to do. “read reviews on the internet of things you might want to buy,’ and “watch 
downloaded or streamed TV shows or movies” (see Fraillon et al. 2014, p. 142). Other items had 
been altered from ICILS 2013 so that it is not possible to make other comparisons. 


The activities that involved acce 
average percentages): 


ssing content from the internet were (in decreasing order of 


e Search for online information about things you are interested in (69%); 


e Use websites, forums, or onli 


ne videos to find out how to do something (50%); 


e Read news stories on the internet (50%); 


e Read reviews on the internet 


of things you might want to buy (39%): and 


e Search the internet to find information about places to go or activities to do (36%). 


The activities that involved accessing content from the internet formed a reliable scale (average 
Cronbach’s alpha across countries = 0.75) representing the frequency with which students 


accessed content from the inte 
F). Students from Kazakhstan 
lowest (Table 5.11). In Denmar 
than female students (i.e., use 


difference between female and 


On the basis of results from thi 
ICT use for accessing content fr 


rnet (see the corresponding item map in Figure F.6 in Appendix 
scored highest on this scale and those from Germany scored 
, Finland, Germany, and Portugal male students scored higher 
the internet more often to access content), but in Korea and 


Kazakhstan the reverse was the case. On average across countries, there was no significant 


male students. 


s scale we concluded that, on average across countries, weekly 
om the internet was higher (by two scale points) for experienced 


computer users than for inexperienced computer users, and higher for those with high levels 


of home computer resources than for those who had low levels of home computer resources 
(by two scale points) (Table 5.12). The difference associated with computer experience was 
significant in all but one of the ICILS 2018 countries (Denmark). The difference associated with 


home computer resources was 
and Germany). 


significant in all but two of the ICILS 2018 countries (Denmark 
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There were small differences on the scale representing the frequency with which students 
accessed content from the internet between students with CIL scores at or above Level 2 and 
those with CIL scores below Level 2 (differences in scale scores averaged one scale point). In 
Korea and Kazakhstan, the differences were a little larger (by four and three scale points). 
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t only one student in five used ICT on a daily basis outside school for school-related 


n contrast, fewer than one student in five used ICT at school for school-related 


purposes and three students in 10 used ICT at school for general purposes. The mismatch 
between ICT engagement out-of-school and ICT engagement in school is wider in some countries 
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Student engagement with ICT for school-related purposes 
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018 student question 
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factors (especially the confidence of teachers in using ICT) on the use of ICT by students but 
noted that the strength of these influences was different in different countries. Data from ICILS 
2013 showed that computer and internet access at school vary across and within countries, and 
are associated with student background and school contexts (Fraillon et al. 2014). 


Extent of student engagement with ICT for school-related purposes 


We asked students to report how often they used ICT for particular school-related purposes 


that ranged from t 


e Prepare reports or essays; 


e Prepare presen 


e Work online with othe 


tations; 


r students; 


he conventional to less conventional. These included the following: 


e Complete worksheets or exercises; 


e Organize your time an 
e Take tests; 


d work; 


e Use software or applications to learn skills or a subject (e.g., mathematics tutoring software, 


anguage 


e Use thei 


e Use codi 


° a 


Although stu 


schoo 
purpo 


most frequently reco 
countries this was repor 


Denm 


reported by seven out o 
relatively few studen 
students reported using the internet 


ke vid 


| day” 


nternet to do 


we reported 


se at 


ark th 


earning software); 


research; 


eo or audio productions. 


the percen 


g software to complete assignments (e.g., Scratch); and 


dents could respond to the question using a set of categories (from “never” to “every 
tages who reported using ICT for a specified school-related 


is was repor 


east once each week? (1 
rded use of ICT 
ted as at least a wee 
ted as awee 
f 10 student 
ts reported usin 


able 5.13). Among the school-related purposes, by far the 
was to “use the internet to do research.” On average across 
ly occurrence by three fifths (59%) of students. In 
ly occurrence by nine tenths (91%) of students. It was also 
s in Portugal (73%) and Uruguay (71%). In some countries 
g the internet to do research. In Finland only 17 percent of 


said that they used the internet to do 


Two of the forms of use of ICT 


for s 


for research on a weekly basis, and in Korea only 36 percent 
research on a weekly basis. 


chool-related purposes concerned how students did their 


work. One of these was organizing their own time and work (in the sense of self-regulation) and 


the other was working online with o 


one quarter of students ( 
for these purposes on a weekly basis. The most common of these uses of ICT on a weekly basis 


were in Denmar 
self-regulation and 4 
was in Germany (9%), fol 
of ICT for collabora 


There was a group of pu 


48% for self-regu 
12% for collabora 


tion was in Finlan 


25% for sel 


tion). The least common wee 
owed by Finland (10%) and Korea (14%). The least common weekly use 


ther students (collaboration). On average across countries 
f-regulation and 28% for collaboration) reported using ICT 


ation and 86% for collaboration) and Kazakhstan (47% for 


ly use of ICT for self-regulation 


rposes liste 


d (9%), followed by Korea (10%) and Germany (12%). 


d that could be considered conventional school activities: 


completing worksheets or exercises (averaging 30%), preparing reports (averaging 26%), preparing 


presentations (averaging 
level of weekly use of ICT 


22%), and taking tests (averaging 20%). Denmark recorded the highest 
| for three of these conventional purposes (61% for preparing reports, 


60% for completing worksheets or exercises, and 45% for preparing presentations). The lowest 


22 In ICILS 2013 we reported use of ICT for each purpose at least once per month. Therefore the two sets of data are 
not comparable. 
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levels of weekly use of ICT for preparing reports were in Finland (7%) and Korea (14%). The 
least widespread weekly use of ICT for completing worksheets were in Finland (6%), Italy (18%), 
and Korea (19%). The lowest prevalence of ICT use for preparing presentations was in Finland 
(7%). Taking tests using ICT on a weekly basis was most common in Kazakhstan (44%) and least 
common in Finland (7%). 


Use of |CT-based software or applications to learn skills or subject content on a weekly basis 
was most common in Denmark (44%) and Kazakhstan (51%) and least common in Finland (12%). 
On average across countries, 24 percent of students reported using these forms of ICT-based 
instructional software on a weekly basis. 


The weekly use of ICT for the two listed specialist purposes was reported by fewer than one in 
five students. Using ICT on a weekly basis for making video or audio productions was reported 
by 18 percent of students on average and the weekly use of coding software such as Scratch 
to complete assignments was reported by 14 percent of students on average across countries. 
High levels of use of ICT for these purposes were reported for Kazakhstan (40% and 27%), and 
very low levels of use were reported for Finland (3% for each form of use). 


We constructed a scale that represented the use of ICT applications as a whole for school- 
related purposes with an average reliability across participating countries of 0.83 (please see the 
corresponding item map in Figure F.7 in Appendix F), where higher scale scores indicated more 
frequent use. We observed that the use of ICT for school-related purposes was, on average 
across countries, slightly greater (by just one scale point) for female than male students (Table 
5.14). ICT use for school-related purposes was notably higher than the ICILS 2018 average in 
Denmark (57 points) and Kazakhstan (56 points), and notably lower in Korea (46 points) and 
Finland (48 points) (Table 5.14). 


We also observed that the use of ICT for school-related purposes was a little higher on average 
for students who were experienced computer users than for other students (by two scale points) 
and alittle higher on average for students who were currently studying a computer-related subject 
than for students who were not (by two scale points) (Table 5.15). The differences associated with 
computer experience were significant in seven of the ICILS 2018 countries, and was greatest 
in Finland (three scale points). The differences in the use of ICT for school-related purposes 
between those who currently studied a computer subject and other students were significant 
inseven ICILS 2018 countries and greatest in Portugal and Kazakhstan. The difference was also 
arge in Moscow (Russian Federation). 


On average across countries, there was no significant difference in the use of ICT for school- 
related purposes between those students who had CIL scores at or above Level 2 and students 
who had CIL scores below Level 2 (Table 5.15). However, there were significant differences 
between these two groups of students in Finland and Korea (in each case by four scale points) in 
favor of those with higher CIL levels. In five ICILS 2018 countries the direction of difference was 
in favor of those with lower CIL levels, and in four countries there was no significant difference. 
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Use of ICT across subject areas 


Research literature over anumber of years has suggested that there are differe 


areas in the extent of use of ICT (Fraillon et al. 2014; Howard et al. 2014). 


asked students how often they used computers during lessons in designated 
areas (“never,’ “in some lessons,’ “in most lessons,’ “in every or almost every 


nces among subject 
In ICILS 2018 we 
subjects or subject 
esson,’ and “| don’t 


study this subject/these subjects”). Student responses inthe last category were treated as missing 
responses. The list of subjects or subject areas that students had to consider was based on a 


list developed for the OECD Teaching and Learning International Study (TAL 


e Language arts: survey language; 


e Language arts: foreign or other national languages; 


e Mathematics; 


S) (OECD 2014): 


e Sciences (general science and/or physics, chemistry, biology, geology, earth sciences); 


e Human sciences or humanities (history, geography, civics, law, economics, etc.); 


e Creative arts (visual arts, music, dance, drama, etc.): 


e Information technology, computer studies, or similar; 


e Practical or vocational studies; and 
e Other. 


We recorded the extent of use of computers during lessons in specified subject areas as the 
percentage of students who reported having used computers in most lessons, or inevery or almost 
every lesson, in that subject area. We found that, on average across countries, the subject area 
with the greatest use of computers was information technology (49%) (Table 5.16). Across the 
subject areas of language arts (27%), sciences (27%), foreign languages (26%), and mathematics 
(25%) there was little variation. The use of computers was a little less in the creative arts (23%). 


The subject areas with the lowest percentages of students who reported using computers in 
most lessons were practical or vocational (19%) and “other” subjects (e.g., moral/ethics, physical 


education, personal and social development) that could not be classified 
subject areas (17%). 


Inthe core subject areas of language arts, mathematics, sciences, and human 
of students in Denmark (69% to 85%). High levels of computer use in these 


these subject areas was reported by around one tenth of the students (8% to 
of computer usage were reported in North Rhine-Westphalia (Germany) (6% 
of computer use in these subject areas were also reported in Luxembourg 
Finland (13% to 18%). 


North Rhine-Westphalia (Germany) (16%). There were relatively high leve 


Finland (11%), and North Rhine-Westphalia (Germany) (9%). 


in the eight listed 


sciences, as well as 


in foreign languages, use of computers in most lessons was reported by an overwhelming majority 


subject areas were 


also reported in Kazakhstan (36% to 45%). In Germany the use of computers in most lessons in 


11%). Similar levels 
to 11%). Low levels 
(16% to 20%) and 


High levels of computer use in the creative arts were reported in Kazakhstan (29%) and Denmark 
(27%). Low levels of computer use in the creative arts were reported in Germany (13%) and 


Is of computer use 


in practical or vocational studies in Kazakhstan (34%) but very low levels in Germany (11%), 


On average across countries, approximately half the students (49%) reported using computers 
in most lessons for information technology, computer studies, or similar. National percentages 


for this indicator were high in Denmark (75%), Portugal (67%), and Uruguay (69%), but low in 


France (18%) and Italy (26%). 
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In ICILS 2018 we asked students how often they used each of 11 listed ICT tools during lessons. 
Students responded using the options: “never,” “in some lessons,’ “in most lessons,” or “in every 
or almost every lesson.’ We recorded the use of each tool in terms of the percentage of students 


who used it in “most” or in “every or almost every lesson” (Table 5.17). Int 


as use in the majority of lessons. 


he text we refer to this 


On average across countries, the most used ICT tools were computer-based information resources 


(e.g., websites, wikis, and encyclo 


software (26%). Use of computer-based information resources in the majority of 
Finland (41%) but not in Germany (11%), 
Use of word processing and presentation software in the 
Denmark (82% and 50% respectively) and Kaza 
15%). Use of word processing software on a weekly basis was 


prevalent in Denmark (59%) and 
(Germany) (12%), and Korea (1 
majority of lessons was notably h 
and 38%) but low in Italy (14% a 


5%). 
fala 
nd 1 


n 


pedias) (29%), word processing software (28%), and presentation 


essons WaS 


orth Rhine-Westphalia 


hstan (39% 


also no 
was on 


Although computer-based information resources were widely 
tive digital learning resources were not. On average ac 
se in a majority of lessons, and there was 


interac 


of students used the 


tably low in Korea (11%). 
y reported by 16 percen 


Similar 


y, there was 


ow usage of specialist tools even thou 
productivity software. An average of only 14 percent of stud 


Of 
t of students on average across 


gh 


productivity tools, spreadsheet use in the majori 


ty of lessons 
CILS 2018 countries. 


used on the majority of lessons, 
ross countries, only 15 percent 


ittle variation among countries. 
there was high use of common 
ents used graphing or drawing 


software in a majority of lessons and only 11 percent of stu 
iting, web production) in a majority of lessons. 


tools (e.g., media capture and ed 


There were several tools listed 
used of these were tutorial s 
countries, 13 percent of stude 
students reported using concept 
and modeling software, in a maj 
used in a majority of lessons by 


From the responses to these i 
represented the exten 
presentation software a 


applications (multimedia produc 
modeling software, computer-ba 
and graphing or drawing softwa 
count 
F.8.an 


We found that general applica 


that could be categorized as 
oftware or practice program 
nts used these tools in 
m 
ity of lesson 


tem 
t to which general a 


apping sof 
s. Tools 


y 10 percent of stu 


s we developed 


pplications (producti 


nd compu 
Cronbach's alpha across countries was 0.72). Theo 
tion, concept map 
sed information resources, interacti 
re) were used dur 
ries was 0.84). Tutorial software and spreadsh 
d F.9 in Appendix F contain the two item maps for these two scales.) 


ter-based information resources 


ping, real-world d 


two scales (Table 5.18 
vity, word process 


were 


ther represented the ext 
ata captu 


ve digi 


eets were not inc 


udedi 


tions were used in class to a greater ex 


dents used multimedia production 


earning tools. The most frequently 
s. On average across ICILS 2018 
a majority of lessons. Just nine percent of 
tware, and eight percent reported using simulations 
for digitally capturing real-world data were 
dents on average. 


. One of these scales 
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The use of general applications in class was slightly more frequently reported by female than male 
students, a difference which was statistically significant overall as well as in several individual 
countries (Table 5.19). In Germany, male students reported greater use of general applications in 
class than did female students. Using general applications in class was more frequently reported 
by students who were currently studying ICT than by students who were not. On average the 
difference was three scale points and the difference was four or more scale points in Chile, 
Kazakhstan, and Portugal. The difference was also large in the benchmarking participants: 

oscow (Russian Federation) and North Rhine-Westphalia (Germany). 


Use of specialist applications in class was more frequently reported by male students than by 
female students overall (by an average of two scale points), and the difference was significant 
in eight of the 11 ICILS 2018 countries (Table 5.20). In the remaining five ICILS 2018 countries 
the differences were not significant. However, the use of specialist applications in class was 
more frequently reported by students who were studying ICT than by those who were not (by 
an average of three scale points) in all except two ICILS 2018 countries. Surprisingly, the use of 
specialist applications in class was more frequently reported by students whose CIL score was 
below Level 2 than by students whose CIL scores were at or above Level 2. This unexpected 
result deserves further investigation, including the possibility that it might reflect differences in 
the types of subjects studied by low and high achieving students. 


Inferences 


CT use for school-related purposes varies according to context. It appears to depend on the 
extent to which ICT is embedded in national curricula and pedagogy. The extent to which students 
use ICT for school-related purposes was higher than the ICILS 2018 average in Denmark and 
Kazakhstan and lower in Korea and Finland. ICT use for school-related pruposes also varied 
across subject areas: the greatest use occurred in foreign languages and the sciences and the 
owest use of computers occurred in practical or vocational studies. ICT use for school-related 
purposes was also associated with student attributes. It was a little higher on average for students 
who were experienced computer users than those who were less-experienced computer users 
and was higher for students who were currently studying a computer-related subject than for 
students who were not. 


ICT use for school-related purposes predominantly involved general applications. The most 
frequent reported school-related use of ICT among grade 8 students was for doing research on 
the internet. Approximately three students in five reported doing this at least once per week 
using computer-based information resources. About one quarter of the students used ICT in 
class on a weekly basis to create and edit documents (i.e., prepare reports and essays) using 
word processing and presentation software. Lower percentages of students indicating the use 
of applications using the potential of ICT to transform pedagogy (such as concept mapping 
software, simulations and modeling software, or digitally capturing real-world data). These findings 
may suggest that there remains a challenge to make use of the full potential of ICT in schools. 
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Learning about ICT at school 
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At the beginning of this chapter, we noted that opportunity to learn referred to the time allocated 
assessed and the curriculum content that was 
the focus of that time (Scheerens 2017). In ICILS 2018 the concepts being assessed were CIL 


for students to be taught the concepts being 


and CT. Although it was not possible to measure the time al 


ocated to teaching CIL and CT, 


because they were sometimes taught in several curriculum areas, it was possible to ask students 
out these two dimensions of ICT. In the student 


to indicate the emphasis placed on learning ab 


questionnaire we asked students to indicate the extent to wh 


ich they had learned (“to a large 


extent,’ “to a moderate extent,’ “to a small extent,’ “not at all”) how to do various ICT tasks. 


Learning about CIL at school 


The ICT tasks that we took as being concerned 


e Provide references to internet sources; 


e Search for information using ICT; 


with CIL were: 


e Present information for a given audience or purpose using ICT, 


e Work out whether to trust information from 


the internet; 


e Decide what information obtained from the internet is relevant to include in school work; 


e Organize information obtai 
e Decide where to look for in 
e Use ICT to collaborate with 


ned from internet sources; 


others. 


formation on the internet about an unfamiliar topic; and 


We examined the percentages recording that th 
extent (Table 5.21). These data indicated small 
60 percent for “use ICT to collaborate with o 


using ICT.’ Overall, the results suggested that 
of CIL at school. The percentage of students was notably high in 
“organize information obtained from internet 


(Russian Federation). The percentages were ge 


what we had observed in the freque 
Portugal, and 


scores on this scale (Table 5.22). Di 


n order to explore differences among groups of students 
earning of CIL tasks, we derived a scale based on student responses to the eight aspects of 
CIL shown above. The scale had a reliability of 0.88 (Cronbach's alpha) o 
countries (the item map for this scale is shown in Figure F.10 in Append 
the scale indicate greater attribution to school-based CIL learni 


oscow (Russian Fede 
France, Germany, Luxembourg, and 


ncy distributions for item 


orth Rhine-Westpha 
fferences between fema 


which they attributed their CIL learning to school instruction 


point in favor of female students, o 
(but less than two scale points) in fi 


students scored higher (by just under two scale points) than male studen 


nN average across countri 


ve of the ICILS 2018 co 


ration), recorded notably high avera 


ey learned about CIL tasks to alarge or moderate 
variations across the various tasks, ranging from 
thers” to 74 percent for “search for information 
students learn about constituent components 
Denmark (for all tasks except 
sources’), Kazakhstan, Portugal, and Moscow 
nerally low in France, Luxembourg, and Germany. 


in countries, in students’ reported 


s. Students in 
ia (Germany 


were very sm 
es, and being 


untries. In Ch 


n average across ICILS 
ix F). 
ng. The scale scores confirmed 
Denmark, Kazakhstan, 
ge scores on this scale. 
recorded notably low 
e and male students in the extent to 
all, being only one scale 
statistically significant 
ile and Finland, female 
ts. 


Higher scores on 


The differences in the CIL learning scale scores between students with five or more years of 
computer experience and those with less than five years of computer experience were significant 
and positive in six countries but were small (Table 5.23). The largest difference was three scale 
points for Korea. In Uruguay the direction of the difference was in the reverse direction. On 
average across countries, there was no difference associated with computer experience. Not 
surprisingly, the CIL learning scale scores were higher for students currently studying ICT subjects 
than for those who were not. This difference was significant and positive in eight ICILS 2018 
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countries and averaged two scale points overall. The difference was large in Portugal (six points) 
and Uruguay (six points). In Denmark the difference was in the opposite direction. 


CIL learning at school scale scores were significantly higher for students with CIL scores at 
or above Level 2 than for students with CIL scores below Level 2 overall in eight ICILS 2018 
countries as well as Moscow (Russian Federation) (Table 5.23). On average across ICILS 2018 
countries the difference was two scale points and in Denmark the difference was four scale 
points. In Uruguay there was a small difference in the reverse direction. 


Learning about CT at school 


Inthe student questionnaire we asked students to indicate the extent to which they had 
learned how to do various CT-related tasks at school. The tasks were: 


e Display information in different ways; 
e Break acomplex process into smaller parts; 


e Understand diagrams that describe or show real-world problems; 


e Plan tasks by setting out the steps needed to complete them; 

e Use tools to make diagrams that help to solve problems; 

e Use simulations to help understand or solve real-world problems; 
e Make flow diagrams to show the different parts of a process; 


e Record and evaluate data to understand and solve a problem; and 


e Use real-world data to solve and revise solutions to problems. 


We examined the percentages of students who reported having learned aspects of CT to a 
large or moderate extent (Table 5.24). These data indicated variations across the aspects of CT 
ranging from 45 percent for “make flow diagrams to show the different parts of a process” to 76 
percent for “display information in different ways.” Overall, the results suggested that students 
had learned about aspects of CT at school. However, there appeared to be larger differences 
among countries for learning about CT than had been the case for learning about CIL. The 
percentages of students were notably higher on average in Kazakhstan, Chile, and Uruguay 
as well as in Denmark on some aspects of CT. The percentages of students were also high in 
Moscow (Russian Federation). The percentages were generally low across these eight tasks in 
Luxembourg, Germany, North Rhine-Westphalia (Germany), as well as in France and Portuga 
on some aspects of CT. 


n order to explore differences in countries and among groups of students in countries, in 
students’ reported learning of CT-related tasks, we derived a scale based on student reports of 
earning about aspects of CT with an average Cronbach’s alpha across countries of 0.90 (Figure 
F.11 in Appendix F shows the corresponding item map for this scale). Higher scores on the scale 
indicate greater attribution to school for learning about CT. The scale scores confirmed what 
we had observed in the frequency distributions for items. Students in Kazakhstan, Chile, and 
Uruguay, as well as Moscow (Russian Federation), recorded high average scores on this scale 
but Germany and Luxembourg recorded low scores on this scale (Table 5.25). 
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Differences between female and male students in the extent to which they attributed their CT- 
related learning to school instruction were very small, being less than one scale point in favor of 
male students, on average across countries, and being statistically significant in seven ICILS 2018 
countries (Table 5.25). Although the differences were small, the overall pattern of differences 
was the converse to that found for students’ reports of learning CIL in which female students 
tended to report more than male students that they had learned about CIL in school (Table 5.22). 


Students with five or more years of computer experience had significantly higher scale scores 
than those with less than five years of experience in just three ICILS 2018 countries (Italy, 
Kazakhstan, and Denmark). However, in North Rhine-Westphalia (Germany) there was a small 
difference in a negative direction (Table 5.26). The CT learning scale scores were higher for 
students currently studying ICT subjects than for those who were not. This difference was 
significant in all countries except Uruguay and averaged three scale points overall. Among ICILS 
2018 countries the difference was largest in Portugal (four scale points); however, in Moscow 
(Russian Federation) the difference was six scale points. 


Student perceptions of ICT 


We investigated two main aspects of students’ perceptions of ICT as part of the broad field 
of emotional engagement with ICT. The first aspect was students’ perceptions of themselves 
in relation to ICT: ICT self-efficacy. We asked students to indicate how well they felt that they 
could accomplish various ICT tasks. Based on the results from ICILS 2013 (Fraillon et al. 2014) 
we formed two constructs from these tasks. The first referred to ICT self-efficacy in relation 
to common productivity applications (typically embodied in office applications) and the second 
referred to ICT self-efficacy in relation to specialist tasks (such as coding, database management, 
and webpage construction). 


The second aspect of students’ emotional engagement with ICT was their attitudes to ICT in their 
futures. We asked about the extent to which they saw aspects of ICT as beneficial for society, 
the extent to which they saw aspects of ICT as detrimental for society. Our conception of ICT 
societal futures envisaged these as separate dimensions rather than as simple polar opposites. 
According to this conception it was possible to envisage some aspects of ICT as beneficial for 
society and other aspects as detrimental to society. We also asked students about the extent 
to which they saw ICT as important for their personal futures. 


ICT self-efficacy 


The concept of self-efficacy refers to an individual’s belief in their capacity to organize and 
execute a course of action to obtain specific outcomes (Bandura 1997). This, in turn, influences 
their choices with regard to undertaking tasks, the effort they expend on them, and the extent to 
which they persevere with a task. In ICILS 2013, we invoked two constructs that referenced ICT 
self-efficacy: ICT self-efficacy regarding the use of general applications and ICT self-efficacy regarding 
the use of specialist applications. In ICILS 2013 we referred to these constructs as ICT self-efficacy 
in basic ICT skills and ICT self-efficacy in advanced ICT skills. |CILS 2013 found that ICT self-efficacy 
in basic ICT skills, which was based on student confidence in undertaking general |CT-based tasks 
such as creating or editing documents, or searching and finding information on the internet, was 
positively associated with CIL. However, ICT self-efficacy in advanced ICT skills, which was based 
on student confidence to carry out tasks such as building or editing a webpage, or creating a 
computer program or macro, was not associated with CIL (Fraillon et al. 2014; Rohatgi et al. 2016). 


As part of the ICILS 2018 student questionnaire we asked students to indicate how well they 
thought they could do each of 13 ICT-based tasks. The response categories were “I know how 
to do this,” “| have never done this but | could work out how to do this,’ and “| do not think | 
could do this.” For the purposes of analyses at the item level, we collapsed the second and third 
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categories and gave the first category a score of one and the second a score of zero. 


The tasks listed were (in order of increasing difficulty): 


The percentages of students who reported that they 


Search for and find relevant information for a school project on the internet; 


Insert an image into a document or message; 


Install a program or app; 


Write or edit text for a school assignment; 


Upload text, images, or video to ano 


nline profile; 


Edit digital photographs or other graphic images; 


Judge whether you can trust information you find on the internet; 


Create a multimedia presentation (with sound, pictures, or video); 


Change the settings on your device to improve the way it operates; 


Set up a local area network of computers or other ICT; 


Build or edit a webpage; 


Create a database (e.g., using Microsoft Access®); and 


Create a computer program, macro, 


which reflect how difficult students pe 


a 


the internet”) 


computer program or macro’) to 88 percent 
Table 5.27). There were also dif 


or app (e.g., in Basic, Visual Basic). 


knew how to do these tasks by themselves, 


rceived each task to be, ranged from 18 percent (“create 


fferences among countries 


of 10 students in Denmark indicated that they could search for and find 


for a school pro 


ject on the internet (95%), wri 


insert an image into a document or message (94%). 


We for 


med two scales based on these item 


(“search for and find information you need on 


. More than nine out 
relevant information 


te or edit text for a school assignment (94%), and 


s in order to explore across-country and other 


differences in students’ ICT self-efficacy. One of those scales (based on eight items) reflected 
students’ ICT self-efficacy regarding the use of general applications (coefficient alpha = 0.83). The 
other (based on four items) was related to students’ ICT self-efficacy regarding the use of specialist 
applications?? (coefficient alpha = 0.73). (The corresponding item maps are shown in Figures 
6.11 and F.13 in Appendix F.) 


There were only small differences among countries on these scales, although the mean score 
onthe ICT self-efficacy (general applications) scale for Kazakhstan was low (45) and for Portugal 


was high (53) (Table 5.28). Interestin 
applications) scale for Kazakhstan was 


Wn 


Table 5.29). The scores for female stud 
for male students. There were significant differences in ICT self-efficacy (general applications) 
n every country and overall by three scale points in favor 
ter for five or more years compared with those who had 


associated with computer experience i 
O 
ess than five years of experience. 


Tt 


tatistically significant gender differe 


ttle difference in the ICT self-efficacy 


gly, the mean score on the ICT se 


nces in ICT self-efficacy (general a 


female students, emerged in Korea, Chile, and Kazakhstan. However, on average, there was 
(general applications) scores of female and male students 


high (53) and the mean score for Denmark was low (47). 


f-efficacy (specialist 


pplications), favoring 


f those who had been using a compu 


ents were, on average, two scale poi 


nts higher than those 


here was a substantial difference in ICT self-efficacy (general applications) between those with 


CIL scores at or above Level 2 and those with CIL scores below Level 2. On average students 


with high CIL scores had ICT self-ef 


23 One of the items (change the settings on 


scores because it did not fit with either of the 


ICT self-efficacy scales. 


ficacy regarding the use of general applications that was five 


your device to improve the way it operates) was not used in calculating scale 
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scale score points higher t 
was nine scale points in Ko 


han among students with low CIL scores (Table 5.29). The difference 
rea. 


There were significant gender differen 
applications favoring male 
points (Tab 
Westphalia 


The differences associated with computer experience were much more closely aligned to what 
we expected. The ICT self-efficacy (specialist applications) scores of students with five or more 
years of computer experience were, on average across countries, two points higher than the 
ICT self-efficacy (specialist applications) scores of students with less than five years of computer 
experience (Table 5.30). The difference was largest in Germany, North Rhine-Westphalia 
(Germany), Luxembourg, and Italy (three points). 


ces in ICT self-efficacy regarding the use of specialist 
students in all countries. On average the difference was four scale 
e 5.30). The gap was large in Denmark (eight points), Germany, and North Rhin 
(Germany) (both six points). 


e- 


There was little systematic difference between the ICT self-efficacy (specialist appli 
of students and their CIL score. Those with CIL scores at or above Level 2 had higher specialist 
ICT self-efficacy scores than those with CIL scores below Level 2 in two countries, but the 
reverse was observed in six countries and there was no significant difference in five countries. 


cations) scores 


Attitudes to ICT in society 


We asked students to indica 
them with a set of 11 stateme 
Students responded to these i 
disagree”). We reported 
who strongly agreed or a 


te their attitudes to the val 
nts that balanced positive an 
tems using four response cat 
the percentage agreement for eac 
greed with the statement. 


ue of ICT in society. We presented 
d negative views of ICT (Table 5.31). 
egories (“strongly agree” to “strongly 
h item by combining the percentages 


There were high percentages who expressed agreement with statements referring to positive 
outcomes of ICT for society such as “ICT helps us to understand the world better” (86%), “advances 
in ICT usually improve people’s living conditions” (85%), “ICT is valuable to society” (84%), and 
“advances in ICT bring many social benefits” (83%). On the other hand there were moderately 
high percentages who expressed reservations by recording agreements with statements referring 
to negative outcomes of ICT for society such as “people spend far too much time using ICT” 
(80%), “using ICT may be dangerous for people’s health” (69%), “using ICT makes people more 
isolated in society” (66%), and “with more ICT there will be fewer jobs” (52%). 


There were also three items that were concer 
and study: “learning how to use ICT applicatio 
(68%), “I hope to find a job that involves specia 
related to ICT after secondary school” (49%). 


ned with expectations of future ICT use for work 
ns will help me to do the work | am interested in” 
ist ICT” (51%), and “I would like to study subjects 


We did not observe large differences among countries and constructed th 
perceptions of positive outcomes of ICT for society (average coefficient al 
of negative outcomes of ICT for society (average coefficien 
differences among countries and subgroups, expectations of fu 
(coefficient alpha = 0.80) (see Figures F.14, F.15, and F.17 in 
items maps). 


ree scales representing: 
pha = 0.75); perceptions 
t alpha = 0.66); and, to explore 
ture ICT use for work and study 
Appendix F for corresponding 


We observed stronger support for positive ou 
Moscow (Russian Federation) and less strong s 


Luxembourg (Table 5.32). We also observed stronger support for negative outcomes of ICT for 
ly 


society in Chile and Uruguay and less strong s 


in Finland, Denmark, and Moscow (Russian Fed 


Stronger support for positive outcomes of ICT 


tcomes of ICT for society in Korea, Portugal, an 
upport for positive outcomes of ICT for society | 


upport for negative outcomes of ICT for socie 
eration). 


d 
n 


for society was expressed by male than by fema 
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students in all countries (Table 5.33). On average across ICILS 2018 countries, the difference 
was three scale points, but in Germany, North Rhine-Westphalia (Germany), and Denmark the 
ifference was six scale points. 


d 

Students with five or more years of computer experience expressed significantly more strongly 
positive views of ICT for society than students with less than five years of computer experience, in 
all countries (Table 5.33). On average across countries, the difference was two scale points. More 
positive views of ICT for society were expressed by students with ClL scores at Level 2 or above 
in just five ICILS 2018 countries, and the average difference was two scale points. 


More strongly negative views of ICT for society were expressed by female students than by male 
students in eight ICILS 2018 countries (Table 5.34). On average across countries the difference was 
two scale points. The difference between male and female students was largest in Germany (the 
difference was three scale points). There was no significant difference in the strength of negative 
perceptions of ICT for society between male and female students in Chile, Kazakhstan, and Uruguay. 


There were few significant differences, and no difference on average across countries, in the 
strength of negative perceptions of ICT for society associated with computer experience or CIL 
(Table 5.34). In Chile more experienced computer users expressed negative views of ICT for society 
more strongly than other students and in Denmark less experienced computer users expressed 
negative views of ICT for society more strongly than other students. 


There was no overall difference in the strength of negative perceptions of ICT for society between 
students with CIL scores at Level 2 or above and students with CIL scores below Level 2 (Table 
5.34). Among ICILS 2018 countries, only in Portugal was there a significant difference with more 
strongly negative views expressed by students with lower CIL. In Moscow (Russian Federation) 
students with lower CIL scores were also more likely to have negative perceptions of ICT for 
society (a scale difference of two points). 


The scale scores for student expectations of future ICT use in work and study differed among 
countries, with students in Uruguay expressing the most positive scores and students in Denmark 
expressing the least positive scores (Table 5.35). The difference between these two countries 
was seven scale points. In all countries male students recorded significantly higher expectations 
of future ICT use than did female students. On average across countries the difference was five 
scale points (Table 5.35). 


Associations of students’ ICT self-efficacy with CIL and CT 


Students’ ICT self-efficacy (general applications) scores were significantly and moderately 
correlated with CIL in all countries (Table 5.36). On average across countries, the correlation 
coefficient was 0.32 and it ranged from 0.24 (Portugal) to 0.38 (Italy). In contrast there was little 
correlation between ICT self-efficacy (specialist applications) scores and CIL. Inseven ICILS 2018 
countries the correlation coefficient was negative but small, in two countries it was positive but 
small, and in two countries it was not significant. On average the correlation coefficient was -0.04. 


Students’ CT was also significantly but moderately correlated with ICT self-efficacy (general 
applications) scores in all seven participating ICILS 2018 countries. On average across countries 
the correlation coefficient was 0.26 (Table 5.36). Again there were weak correlations of CT 
and ICT with ICT self-efficacy (specialist applications) scores. On average across countries, the 
correlation coefficient was -0.04. The correlation coefficient was always small and was significant 
and negative in three countries, significant and positive in one country, and not significant in 
the remaining three countries. 
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Table 5.36: Correlation coefficients of students’ ICT self-efficacy for both general applications and specialist applications 


Notes: Standard errors appear in parentheses. Statistically significant coefficients (pp < 0.05) are shown in bold. 


+ Met guidelines for sampling participation rates only after replacement schools were included. 


tt Nearly met guidelines for sampling participation rates after replacement schools were included. 
1 National defined population covers 90% to 95% of the national target population. 
Country surveyed target grade in the first half of the school year. 


with CIL and CT 
Country Correlation of CIL with: Correlation of CT with: 
Students’ ICT Students’ ICT Students’ ICT Students’ ICT 
self-efficacy self-efficacy self-efficacy self-efficacy 
regarding the use of regarding the use of regarding the use of regarding the use of 
general applications | specialist applications | general applications | specialist applications 

Chile 0.33 (0.03) -0.06 (0.03 
Denmarkt * 0.33 (0.03) -0.03 (0.03 0.30 (0.02) 0.02 (0.03) 
Finland 0.33 (0.02) -0.06 (0.02 0.30 (0.02) -0.03 (0.02) 
France 0.25 (0.02) -0.09 (0.02 0.19 (0.02) -0.11 (0.03) 
Germany 0.26 (0.03) -0.07 (0.03 0.18 (0.03) -0.03 (0.03) 
taly* 0.38 (0.02) 0.00 (0.02 

azakhstan! 0.34 (0.03) 0.12 (0.03 

orea, Republic of 0.48 (0.02) 0.15 (0.02 0.39 (0.02) 0.10 (0.02 
Luxembourg 0.28 (0.01) -0.10 (0.01 0.24 (0.01) -0.08 (0.01 
Portugaltt * 0.24 (0.03) -0.15 (0.03 0.21 (0.03) -0.16 (0.03 
Uruguay 0.34 (0.02) -0.11 (0.03 
ICILS 2018 average 0.32 (0.01) -0.04 (0.01 0.26 (0.01) -0.04 (0.01 
Not meeting sample participation requirements 
United States 0.39 (0.01) 0.01 (0.01 0.31 (0.01) -0.01 (0.02 
Benchmarking participants meeting sample participation requirements 
Moscow (Russian Federation) 0.27 (0.03) -0.03 (0.03 
North Rhine-Westphalia (Germany) 0.26 (0.03) -0.03 (0.03 0.17 (0.03) -0.01 (0.03) 
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CHAPTER 6: 


Teaching with and about information and 
communications technologies 


Chapter highlights 


On average across countries, most teachers tended to have experience and familiarity with 


information and communications technology (ICT). 


More than two thirds of teachers on average had at least five years of experience with the 
use of ICT during lessons or with their preparation. (Table 6.1) 
While ICT use was quite widespread among teachers in general, more frequent use when 
teaching was only reported by less than half of the teachers, with considerable variation 
across participating countries. (Table 6.1) 


Most teachers tended to recognize the value of ICT use for teaching and were confident 
about using this technology. 


While majorities among teachers across countries expressed confidence when undertaking 
a large number of different |CT-related tasks, they lacked confidence regarding the use 
of online discussions, online collaboration, and the use of learning management systems. 
(Table 6.2 


Older teachers felt less confident in using ICT (Table 6.3) and teachers who used ICT 
more frequently for teaching were more confident about using ICT. (Table 6.7) 


Large majorities among teachers endorsed the advantages of ICT use for student learning 
(Table 6.4), but there were also considerable proportions of teachers mindful of negative 
effects. (Table 6.5) 
In most countries, teachers with higher levels of endorsement of positive outcomes 
when using ICT for teaching and those who reported daily ICT use for teaching were 
less concerned about its potential negative effects. (Table 6.6) 


There were considerable differences across countries in the availability of ICT at schools, 
the extent of teacher collaboration, and conditions for professional learning. 


ICILS 2018 school and teacher data show large differences in the availability and 
appropriateness of ICT resources across countries. (Table 6.8, Table 6.9, and Table 6.10) 
Majorities among teachers perceived collaboration between teachers (Table 6.11) and 
those who reported a greater extent of collaboration also perceived higher levels of ICT 
availability at their schools. (Table 6.12) 


Teachers who used ICT more frequently in class were also more positive about teacher 
collaboration and ICT resources at school. (Table 6.13) 


Teacher and school-level reports suggest large differences between countries in the 
expectations and requirements with regard to teacher qualifications for ICT use as well 
as the participation in professional learning activities. (Table 6.14 and Table 6.15) 
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Teachers’ placed some or strong emphasis on developing skills related to computer and 
information literacy and computational thinking. 


e There was considerable variation in teacher emphasis across different subject areas. 
(Table 6.16 and Table 6.18) 


e Teachers’ confidence in using ICT, positive perceptions of pedagogical ICT use, and 
positive perceptions of teacher collaboration were positively associated with teacher 
emphasis on student learning of computer and information literacy and coding-related 
tasks. (Table 6.17 and Table 6.19) 


Teachers’ use of ICT in classrooms was still limited and depended on complexity of tasks 
and applications. 


e Teachers tended to report higher levels of use for general utility tools than for digital 
learning tools. (Table 6.20 and Table 6.21) 


e Teachers’ use of ICT for student activities and teaching practices varies both across the 
different types of activities/practices as well as considerably across countries. (Table 
6.22 and Table 6.23) 
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Introduction 


The International Computer and Information Literacy Study (ICILS) 2018 contextual framework 
(Fraillon et al. 2019) assumes that computer and information literacy (CIL) and computational 
thinking (CT) are influenced by context factors that are either antecedents (e.g., school resources) 
or process-related (e.g., teaching practices) and can be located at different levels including in the 
wider community, schools and classrooms, home and peer contexts, and the individual student 
level. Further details of this conceptual model are provided in Chapter 1. In this chapter we will 
focus on antecedent and process-related contextual variables that are located at the level of 
schools and classrooms. 


Analyses of students’ acquisition of CIL and CT need to acknowledge the key role that schoo 
and classrooms play, and in this context it is interesting to note that results from a meta-analysi 
highlighted the positive impact of the use of information and communications technology (ICT) 
on classroom achievement (Tamin et al. 2011). In recent decades there has been a considerable 
increase in an educational policy focus on providing schools and teachers with ICT for use in 

e 

S 


S 
i) 


education. However, results from cross-national studies have repeatedly illustrated that ther 
are considerable differences across countries, across different subject areas, and across schoo 
within countries, as well as between different subgroups of teachers, in particular related to age 
(Fraillon et al. 2014; Kozma 2003; Lawet al. 2008). Research has also emphasized the importance 
of system and school factors in supporting teachers’ pedagogical use of ICT (Eickelmann 2011; 
Gerick et al. 2017). 


Inthis chapter we report the results of analyses related to teaching with and about ICT at schools 
across participating countries. It is based on data derived from the teacher, school principal, 
and ICT coordinator questionnaires, and addresses Research Question 2 related to CIL and CT: 
What aspects of schools and countries are related to students’ CIL/CT? To this end, the chapter 
reviews variation in approaches to and variation in the use and availability of ICT across different 
national contexts. It also focuses on teachers’ familiarity with ICT, their views regarding its use for 
teaching and learning, and the extent to which it is used in classrooms. Furthermore, it reviews 
the emphasis teachers place on developing CIL and CT across different subject areas and their 
relationship with the actual use of ICT in lessons. 


The ICILS teacher survey gathered data from all teachers at sampled schools who taught at the 
target grade in each country (typically grade 8). Samples of 15 teachers were selected at random 
in each participating school and invited to participate in the survey. In schools with 20 or fewer 
teachers, all of them were invited to complete the questionnaire. 


To capture teachers’ |CT-related perceptions in their in-class teaching, |CILS 2018 asked teachers 
some questions with regard to a specific reference class taught at the target grade. Teachers 
were asked to identify the reference class as the first class they taught on the Tuesday (or on the 
next day in the week they were teaching if they did not have classes on Tuesday) following the 
last weekend. A large number of questions were then asked about that particular reference class. 


ICILS 2018 also collected data at the school level using two instruments. School principals 
completed a 10-minute questionnaire providing information about school characteristics, school 
approaches to ClL and CT learning, and incorporating ICT into teaching and learning. Schools were 
also asked to name an ICT coordinator, this could be either a formal or informal position or be 
another specifically nominated staff member with knowledge about ICT-related issues at school. 
These staff members completed a 10-minute questionnaire providing information about ICT 
resources, and technical and pedagogical support for ICT use for teaching and learning at school. 
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Data from both of these instruments were reported at the level of students by matching school 
data to individual learners. In line with common practice in other IEA studies, this was done to 
provide a better picture of school contexts for the “average student” in each country as schools 
with few students would otherwise have the same weight as larger schools with much higher 


numbers of students. 


The ICILS 2018 teacher questionnaire was used to measure anumber of constructs that underpin 
scales and items presented in this chapter. We used item response theory (IRT) scaling (Masters 
and Wright 1997; Rasch 1960) to derive the scales presented in this chapter, which were set 
to a metric with a mean of 50 and a standard deviation of 10, with equally weighted national 
data from those countries that had met the IEA sampling requirements for the teacher survey. 
All scales are described in item maps contained in Appendix F of this report. The maps relate 
scale scores to expected item responses under the ICILS scaling model (also illustrated in Figure 
F.1, Appendix F). Greater detail regarding the scaling procedures for questionnaire items will be 


provided in the ICILS 2018 technical report (Fraillon et al. 2020). 


In five countries the teacher participation rates 


did not meet the IEA requirements for sampling 


participation. Therefore, in adherence to IEA standards for reporting, teacher questionnaire results 
from this relatively large group of countries are reported in an extra section of the respective 
reporting tables and not included in comparisons with those from other countries. Readers 
are advised to interpret teacher data from countries not meeting the sample participation 


requirements with due caution. 


Teachers’ familiarity with and views of ICT 


Teachers’ experience and use 


using ICT at school when teaching, at school for 


Prior research has indicated relationships between experiences of teaching staff with ICT, the 
extent of its use, and their attitudes toward digital technologies as tools for teaching and learning 
(see Drossel et al. 2017; Fraillon et al. 2014; Nikolopolou and Gialamas 2016). The ICILS 2018 
teacher questionnaire asked teachers about their (approximate) years of experience (“never 
“less than two years,” “between two and five years,” or “more than five years”) with using ICT 
for teaching purposes during lessons as well as when preparing lessons. Given that majorities 
of teachers across all countries were found in the highest categories of years of experience, we 
present the results in percentages of teachers reporting to have five or more years of experience. 


Furthermore, teachers were asked to rate their frequency (“never,’ “less than once a month, “at 
east once a month but not weekly,’ “at least once a week but not every day,’ or “every day”) of 


other work-related purposes, outside school for 


work-related purposes, and outside school for non-work-related purposes. Results are presented 


as percentages of teachers who reported using 


CT every day. 


On average across participating countries, about two thirds of teachers reported having five or 
more years of experience with the use of ICT during lessons, while an even higher percentage (72%) 
reported having asimilar length of experience with the use of ICT for preparing lessons (Table 6.1). 
The highest percentages of teachers indicating long experience for both activities were recorded 
in Denmark, Portugal, and the benchmarking participant Moscow (Russian Federation), while the 
lowest percentages were reported in Italy, Kazakhstan, and the benchmarking participant North 


Rhine-Westphalia (Germany). 
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On average across countries, two thirds of teachers used ICT daily for other work-related purposes 
at school while less than half of the teachers reported daily use of ICT when teaching. The highest 
percentages of ICT for teaching were found among teachers in Denmark, Finland, and Moscow 
(Russian Federation), while the lowest percentages of this type of ICT use were recorded in Chile, 
taly, and North Rhine-Westphalia (Germany). In most countries, more than half of the teachers 
reported daily use of ICT for work-related purposes at school; however, in the benchmarking 
participant North Rhine-Westphalia (Germany) we found very low percentages, with only every 
fifth teacher reporting use of ICT for these purposes. 


Percentages of at least weekly ICT use for work-related purposes tended to be above 50 percent 
in most countries; however, among teachers in Korea only about a quarter fell into this category. 
Using ICT at least weekly for non-work-related purposes was reported by most teachers across 
countries; however, somewhat lower percentages were recorded in Kazakhstan and Korea. 


Teachers’ ICT-related self-efficacy 


Bandura’s (1997) concept of self-efficacy refers to individuals’ belief in their capacities to organize 
and execute a course of action to obtain specific outcomes, which in turn influence their choices 
with regard to undertaking tasks, the efforts with which they conduct them, and their extent of 
perseveration. There is evidence that teachers’ sense of self-efficacy regarding the use of ICT 
influences its use by teachers in the classroom (Fraillon et al. 2014; Hatlevik 2017; Hatlevik and 
Hatlevik 2018; Law et al. 2008; Nikolopolou and Gialamas 2016). ICILS 2018 asked teachers 
to rate how well they can do a range of different ICT tasks (“I know how to do this,” “| haven't 
done this but | could find out how,’ “I do not think | could do this”). Results are presented as 
the percentages of teachers who reported to know how to do each of these tasks (Table 6.2). 


@ 


On average across participating countries, we found that more than three quarters of teachers 
believed they knew how to “find useful teaching resources on the internet” (95%), “produce 
presentations (e.g., [PowerPoint® or a similar program]), with simple animation functions” (84%), 
“use the internet for online purchases and payments’ (86%), “prepare lessons that involve the use 
of ICT by students” (86%), and “assess student learning” (78%). Somewhat more than two thirds 
expressed confidence in “using a spreadsheet program for keeping records or analyzing data” 
(69%). We observed the lowest levels of confidence (somewhat more than half of the teachers) 
for “contributing to a discussion forum/user group on the internet (e.g., a wiki or blog)” (58%), 
“collaborating with others using shared resources such as Google Docs® or Padlet” (57%), and 
“using a learning management system” (59%). Notably lower levels of confidence were recorded 
in some individual countries, such as for the use of a learning management system in Chile (21%) 
or collaboration with others using shared resources in North Rhine-Westphalia (Germany) (25%). 


All nine items were used to derive an IRT scale reflecting teachers ICT self-efficacy, where higher 
scales reflect higher levels of self-efficacy. The scale had satisfactory reliability (an average 
Cronbach's alpha of 0.80) across participating countries (Figure F.17 in Appendix F contains 
the corresponding item map showing how scale scores relate to item responses). The highest 
levels of teachers’ ICT self-efficacy were observed in Denmark and Portugal, while teachers in 
Italy, Kazakhstan, and North Rhine-Westphalia (Germany) had the lowest levels of teacher self- 
efficacy (Table 6.3). 
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To review differences 


in |CT-related self-efficacy between younger and older teachers, we 


distinguished between teachers who were 40 years of age and older and those who were younger 
than 40 years at the time of the survey. In all participating ICILS 2018 countries we observed 
significantly higher scale scores for teachers who were younger than 40 years compared with 
those who were 40 years of age and older, with an average scale score difference of about four 
ational standard deviation), ranging from a two-point difference in 


points (more than an intern 
Denmark to a six-point d 


example, Fraillon et al. 2014) about the association 
efficacy, as well as illustrating notable differences in 


Teachers’ views of ICT for teaching and learning 


Positive teacher attitudes toward the use of ICT 


fference in Finland. These results confirm earlier findings (see, for 
between age and teacher |CT-related self- 
this association across countries. 


for teaching and learning are regarded as 


important key factors for the implementation of digital technologies for teaching and learning at 
schools (Lawrence and Tar 2018; Tondeur et al. 2017). European survey data from 2011 illustrated 


the classroom (Wastiau et al 


raillon et al. 2014). 


countries there was a widespread recognition o 


that across the region teachers tended to express positive attitudes toward the use of ICT in 


. 2013) and ICILS 2013 data also revealed that across participating 


ICILS 2018 continued to gather teachers’ views on using ICT for 
them about their level of agreement or disagreement (“strong 


f positive benefits from using ICT 


in teaching 


teaching and learning by asking 
y agree,’ “agree,” “disagree,” or 


“strongly disagree’) with a number of statements. Seven of these statements related to positive 
outcomes of using ICT in education that were helping with student learning, while another six 


statements referred to outcomes that reflected potenti 


presented in the following two tables as percentages of 


with each of these statemen 


Across participating countri 
positive outcomes of using 


ts. 


es, large majorities among 


al impediments of 


learning. Results are 


teachers who agreed or strongly agreed 


teachers agreed with statements about 
CT in teaching and learning (Table 6.4). The h 


ighest proportions of 


agreement (above 85%) were recorded for the statements “helps students develop greater interest 


in learning,’ “enables studen 
work at a level appropriate 


75%) were O 
“helps studen 


« 


statements (“helps students develop problem-solving skills” and “enables st 
more effectively”) were endorsed by almost 80 percent across countries. 


When compa 


that on average had the highest 


improvement 


countries. While two thirds or more of teachers expres 


of academic perfor 


ts to access better sources of information,’ and “helps students to 
to their learning needs.’ The lowest proportio 
bserved for the statements “improves academic performance of students” and 
ts develop skills in planning and self-regulation of their wor 


ns on average (below 


” The remaining two 
udents to collaborate 


ring results across countries, there was relatively little variation for the three items 
evels of endorsement. However, the statement relating to the 
mance received varying levels of agreement by teachers across 
sed agreement with this statement in Chile, 


Denmark, Kazakhstan, and Portugal, less than half of the teachers endorsed this notion in Finland 


and North Rhi 


ne-Westphalia (Ge 


rmany). 


On average across participating countries, o 
of using ICT in teaching and learni 


surveyed teachers endorsed the notion that 
however, this was seen as a problem by more t 
half of the teachers across ICILS 2018 countries agreed that ICT use “distracts students from 
learning” (37%), “results in poorer calculation and estimation skills among students” (41%), or 
“limits the amount of personal communication among students” (46%). 


ng, expressi 
from internet sources” (71%) and in “poore 
endorsed by more than half of the teachers (1 


n 


r 


U 


ng that ICT use results in “stud 


y two of the statements about negative outcomes 


written expression among s 


ents copying material 
tudents” (52%), were 


Table 6.5). In most countries, 
sing ICT “impedes concept fo 


ess than a quarter of 
rmation by students;” 


han two thirds of teachers in Kazakhstan. Less than 
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Perceptions of schools’ ICT learning environments 


Perceptions of school resources for ICT use 


Research has suggested that school-level factors related to ICT resourcing and priorities influence 
both the way in which teachers use ICT for teaching and learning, and students’ |CT-related 
learning (Fraillon et al. 2014; Gerick et al. 2017; Lawrence and Tar 2018). In this section, we will 
use data collected from teachers, school principals, and ICT coordinators to review perceptions 
of ICT learning environments at schools in participating ICILS 2018 countries. 


School ICT coordinators were asked about the extent to which they perceived that the use of ICT 
for teaching was hindered by different factors (“a lot,’ “to some extent, “very little” or “not at all”). 
One group of factors related to the lack of computer resources at school, while the other group 
of factors related to pedagogical resources. The results were reported at the level of students: 
that is, in percentages of students who were enrolled at schools where each of the factors was 
reported as a hindrance to ICT use for teaching a lot or to some extent. 


Across countries, almost half of students studied at schools where insufficient bandwidth or speed 
18%), not enough computers for instruction (48%), and lack of sufficiently powerful computers 
17%) were reported as hindrances (Table 6.8). Slightly lower proportions of students were enrolled 
at schools with problems maintaining ICT equipment (44%) and not having enough computer 
software (38%), while on average less than a third of students studied at schools where having 
too few computers with internet connections was reported as a problem. 


There was considerable variation across countries in the percentages of students at schools 
where insufficient computer resources were viewed as a problem. While in Denmark, Korea, 
and Luxembourg only few students were enrolled at schools where computer resources were 
reported as hindrances for using ICT for teaching, this was the case for substantially higher 
proportions of students in Germany, Italy, Portugal, and Uruguay. 


Compared to the results for computer resource hindrances, we observed generally higher 
proportions of students at schools where the lack of pedagogical resources was seen as ahindrance 
to using ICT for teaching across participating countries (Table 6.9). Almost two thirds of students 
studied at schools where insufficient ICT skills among teachers (65%) and insufficient time for 
teachers to prepare lessons (64%) were viewed as hindrances for using ICT for teaching. On 
average, half or more of surveyed students were enrolled at schools where the lack of effective 
professional learning resources for teachers (59%), lack of incentives for teachers to integrate ICT 
in their teaching (57%), and insufficient pedagogic support for the use of ICT (50%) were reported 
as problems. Across countries, lack of an effective online learning support platform at their schools 
affected the relatively lowest proportion of students (44%). 


Again, we observed substantial differences across participating countries. While in Denmark, 
Kazakhstan, Korea, and Moscow (Russian Federation) the proportions of students at schools 
with a reported lack of pedagogical resources tended to be smaller across the different factors, 
these percentages were higher in Finland, Germany, Portugal, and Uruguay. However, it should 
be noted that these patterns were not always consistent across the different factors. For 
example, in most countries there were high proportions of students studying at schools where 
insufficient ICT skills and insufficient time for preparing lessons were regarded as hindering ICT 
use for teaching and learning. 


The teacher questionnaire collected data on teacher perceptions of |CT resources at their schools 
by asking them about their agreement or disagreement with a number of statements (“strongly 
agree, “agree. “disagree,” or “strongly disagree’). Seven of these items (excepting the item “ICT 
is considered a priority for use in teaching”) were used to derive an IRT scale reflecting teachers’ 
perceptions of the availability of ICT resources at school. The scale had satisfactory reliability across 
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participating countries 
greater availability of ICT 


The results at the item 


ing across partici 


PREPARING FOR LIFE INA DIGITAL WORLD 


Cronbach's alpha = 0.87), and higher scores indicate perceptions of 
| resources (see the corresponding item map in Figure F.20 in Appendix 
F linking scale scores to expected item responses). 


evel show that typically more than 80 percent of teachers expressed 
agreement (incorporating “strongly agree” and “agree” responses) that ICT was considered a 
priority for use in teach 
recorded in Italy (74%) and considerably lower percentages found in North Rhine-Westphalia 
(Germany) (33%) (Table 6.10). On average across participat 
also endorsed the statements that their school had sufficient ICT equipment (62%), that their 
school’s computer equipment was up-to-date (61%), that their school had access to sufficient digital 
learning resources (59%), that there was sufficient time to develop expertise in ICT (51%), and that 


pating countries, with somewhat lower proportions 


ng countries, over half of the teachers 


there was sufficient support to maintain ICT resources (55%). Less than half of the teachers, on 
average across countries, agreed that there was enough time to prepare lessons incorporating ICT 
(41%); however, there was some variation regarding this aspect, as two thirds or more of teachers 


in Kazakhstan and Moscow (Russian Federation) regarded time for this as sufficient. 


Research has suggested that there are benefits of teacher collaboration with colleagues for 
the development of ICT self-efficacy and promoting the use of ICT for teaching purposes (see, 
for example, Caspersen and Raaen 2014; 
asked teachers about their perceptions of 


teaching and learni 


disagreement with five statements regardi 


ng at their school. IC 


Drossel et al. 2017; Fraillon et al. 2014). ICILS 2018 
whether and how ICT is used as part of collaborative 
LS 2018 asked teachers about their agreement or 


ng collaboration among colleagues regarding the use 


of ICT (“strongly agree,” “agree,” “disagree,” or “strongly disagree’). All five items were the basis 


for an IRT scale refl 
using ICT. The scale 


ecting teachers’ perceptions of the collaboration between teachers when 
had satisfactory reliability across participating countries (Cronbach's alpha 


= 0.86) and higher scores indicate perceptions of greater collaboration between teachers (see 


the corresponding i 
responses). 


On average across p 


in teaching and learning at their schools (Table 6.11). About three-quarters agreed with th 
statements that they discussed wit 


articipating countries, 


g 
“strongly agree” and “agree” responses) with statements regarding the collaborative use of ICT 
e 
d 


tem map in Figure F.21 in Appendix F linking scale scores to expected item 


h other teachers how to use ICT for teaching topics (75%) an 


teachers tended to express agreement (incorporatin 


that they shared ICT-based resources with other school teachers (74%). Furthermore, about two 
thirds of teachers agreed that they worked together with other teachers on improving the use of 


ICT inclassrooms (65%) and that they observed how other teachers used ICT in teaching (71%). 


The lowest percentages of agreement were recorded for the statement regarding collaboration 


with colleagues to d 


There were some no 
related to teacher co 


evelop ICT-based lessons; however, in most countries more than half the 
teachers expressed agreement (61% on average). 


table differences across countries regarding agreement with the statements 
laboration. While in Kazakhstan and Moscow (Russian Federation) very large 


majorities (above 80%) agreed with all of the statements, there were particularly low proportions 
of agreement in North Rhine-Westphalia (Germany) with the statements regarding working 


together with colleagues on improvements in the use of ICT in lessons (36%) and collaboration 


to develop ICT-base 


When comparing national average scale scores for teachers’ perceptions of the availability of 
ICT resources at school and teachers’ perceptions of collaboration between teachers when using 
ICT, we observed considerable variation across countries for both scales (Table 6.12). When 
looking at the distribution of national averages across countries, it becomes evident that in 


d sessions (30%). 


countries where teachers perceived greater levels of availability of ICT resources, there were also 
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Table 6.11: National percentages of teachers agreein 


PREPARING FOR LIFE INA DIGITAL WORLD 


g with statements about the collaborative use of ICT in teaching and 


learning 
Country Percentage of teachers that agree that: 
| work together with | collaborate with | observe how | discuss with | share ICT-based 
other teachers on colleagues to other teachers other teachers resources with 
improving the use of develop ICT-based use ICT in teaching how to use ICT other teachers in 
ICT in classroom lessons in teaching topics in my school 
teaching 
Chile 57 (1.9) V 59 (1.8) 67 (1.2) V 62 (15) VW 69 (14) V 
Denmark? * 63 (2,2) 57 (20) V S71 75 (4.5) 79 (14) A 
Finland 68 (1.1) A 56 (14) V £5: 42) os 78 (1.0) A 56 (10) W 
Italy? 68 (1.4) 59 (1.8) 74 (1.3) A 80 (1.5) A 73 (1.5) 
Kazakhstan’ 94 (0.9) 93 (0.7) 90 (1.2) A 93 (0.8) A 93 (0.9) A 
Korea, Republic of 50 (1.6) Vv 51 (19) W 69 (13) V 65 (14) W 66 (1.5) V 
Portugal 55 (1.3) VW 55 (i2) V 67 (1.3) V 74 (1.2) 82 (10) A 
ICILS 2018 average 65 (0.6) 61 (0.6) TAKS) iOS 74 (0.5) 
Not meeting teacher sample participation requirements 
France’ 55 (1,7) 44 (1.6) 58 (1.4) 82 (1.1) 64 (1.6) 
Germany 38 (1.8) 33 (1.8) 54 (2.3) 58 (2.0) 54 (2.1) 
Luxembourg 47 (2.1) 39 (2.1) 67 (2.1) 81 (2.0) 70 (1.9) 
United States* 66 (1.5) 59 (1.8) 63 (1.5) 75 (AQ) 74 (1.3) 
Uruguay JO (12) 67 (14) 79 (1.4) 86 (1.0) 78 (1.2) 
Benchmarking participants meeting sample participation requirements 
oscow (Russian Federation) 82 (1.3) A 80 (14) A 91 (08) A 95 (0.5) A 87 (0.8) A 
North Rhine-Westphalia (Germany) 36 (24) W 30 (2.0) W 57 (20) W 67 (1.7) V 61 (1.9) Vv 
Notes: Standard errors appear in parentheses. Because results are rounded to the National ICILS 2018 results are: 
nearest whole number, some totals may appear inconsistent. Comparisons with A More than 10 percentage points above average 
ICILS 2018 only reported for countries or benchmarking participants meeting A. Significantly above average 
sample participation requirements. cn 
: ide ' en aa VY Significantly below average 
Met guidelines for sampling participation rates only after replacement schools V More than 10 percentage points below average 


1 


2 


were included. 


National defined population covers 90% to 95% of the national 
Country surveyed target grade in the first half of the school yea 


teacher perceptions of greater co 


The correlation coefficients w 


indicate that teachers who percei 
to perceive higher levels of teach 
countries was 0.36, ranging from 0.24 (in Finland) to O. 


the associations were statistic 


The highest teacher percepti 


school were recorded in Den 
lowest average was observed 

collaboration, we found the h 
and the lowest average again | 


Earlier, and 
found evidence for an associat 


regarding this technology. In order 
and collaboration were related 


averages be 
reported less frequent use (Ta 


in line with findings fro 


arget population. 
r. 


ithin 


er collaboration. The 


ally significant in all coun 


ns scale scores regardi 
ark, Kazakhstan, and 
North Rhine-Westpha 
ghest averages in Kazak 
n the German benchmar 


O 
mM 
n 


lL, 


ion between teachers’ dai 
to review the extent 


ing participant 
m the previous ICILS cycle in 2013 ( 


average correlation coefficien 


53 (in Moscow, Russian Federati 
tries. 


ng the availability of ICT reso 


laboration between colleagues on the use of ICT at schools. 
The country-level correlation between national average scores on these two scales was 0.65. 
countries (accompanied by their respective standard errors) 
ved greater availability of ICT at their schools also tended 


t across 
on), and 


urces at 


oscow (Russian Federation), while the 
ia (Germany). For perceptions of 


teacher 


stan and Moscow (Russian Fed 


ort 


Fraillon et a 


teac 


y use of ICT for 
to which perceptio 


to the frequency of teach 


tween teachers who reported daily use of ICT for teaching with 


ble 6.13). 


ers’ use of ICT, we compared 


eration) 


h Rhine-Westphalia. 


. 2014), we 
hing and their views 
ns of ICT availability 
scale score 
those teachers who 
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The results demonstrate that across participating countries teachers who used ICT ona daily basis 
for teaching tended to have more positive views of the availability of ICT resources as well as of 
the extent of collaboration between teachers. In most countries the scale score differences were 
statistically significant, except for in Chile regarding the availability of ICT resources at school and 
in Korea regarding teacher collaboration when using ICT. Differences between the comparison 
groups were on average more than three score points for teachers’ perceptions of ICT availability, 
with the largest difference recorded for North Rhine-Westphalia (Germany). Teachers who used 
ICT onadaily basis were also more likely to perceive higher levels of teacher collaboration regarding 
ICT: the average score point difference was almost three points, with the largest differences (above 
five points) recorded in Kazakhstan and North Rhine-Westphalia (Germany). 


These data suggest that there were differences in the perceptions of ICT resources and teacher 
collaboration depending on how often teachers use ICT in class. Readers should note that these 
bivariate associations do not indicate any clear causality. While it is possible that teachers less 
concerned with (and perhaps less knowledgeable of) ICT use tend to perceive ICT environments 
as less favorable, it is also possible that the association is rather explained by the enablement 
of teachers’ use of ICT through higher levels of resources and collaboration with colleagues. 


Perceptions of professional development for pedagogical ICT use 


The development of teacher expertise in |CT-related teaching and learning has been identified as 
an important variable in enabling teachers’ use of ICT and the teaching and learning of |CT-related 
skills (Charalambos and Glass 2007; Law et al. 2008; Lawrence and Tar 2018; Scherer and Siddiq 
2015). The ICILS 2018 school principal questionnaire asked about schools’ expectations and 
requirements of teachers’ acquisition of knowledge with regard to a wider range of ICT-based 
activities. Principals rated the expectation of teachers’ acquisition of knowledge to undertake 
a range of ICT-based activities as “expected and required,” “expected but not required,’ or “not 
expected.” The results are presented as the percentages of students who study at schools where 
principals reported each of the professional development activities as both expected and required. 


The results show very large differences across participating countries (Table 6.14). While in 
Kazakhstan and Moscow (Russian Federation) for all activities, and in Denmark for most activities, 
there were majorities of students who studied at schools where these were required, in most 
countries only relatively few students were enrolled at institutions where teachers were expected 
and required to have the corresponding knowledge for these ICT-related activities. For the 
requirement of integrating ICT into teaching and learning we found the highest proportions of 
students at schools where this was the case (52% on average). Few students across countries 
were enrolled at schools where teachers were expected and required to acquire knowledge about 
the use of e-portfolios for assessment (17%) or about the use of ICT for developing authentic 
assignments for students (18%). 


The ICILS 2018 teacher questionnaire included a question regarding the respondents’ 
participation in a range of different professional learning activities related to ICT over the past 
two years, where respondents were asked to state whether they had done each of these “not 
at all” “once only,’ or “more than once.’ The results are presented as percentages of teachers 
who reported having participated at least once in these activities (i.e., combining percentages 
nthe last two response categories). 


On average across countries, more than half of the surveyed teachers reported having participated 
at least once over the past two years in a course on ICT applications (51%), observed other 
teachers using ICT in teaching (59%), and shared digital teaching and learning resources in a 
digital workspace with others (57%) (Table 6.15). While half of the teachers reported having 
received training in subject-specific digital teaching and learning resources, less than half of 
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them reported participation in courses or webinars on integrating ICT into teaching and learning 
(46%), in |CT-mediated discussions or fora on teaching and learning (40%), or in collaborative 
workspaces to jointly evaluate student work. Less than a third of teachers on average across 


countries reported partici 
or specific learning difficul 


learning of students (28%). 


There were notable variati 


relatively high percentages 


pation in courses on the use of ICT for students with special needs 
ties (24%) and in courses on how to use ICT to support personalized 


ons across participating countries and benchmarking entities. While 
among teachers in Italy, Kazakhstan, and Moscow (Russian Federation) 


tended to report participation in these activities, these proportions were notably smaller in 
Denmark, Portugal, and, in particular, in North Rhine-Westphalia (Germany). It is noteworthy 


that some of the countries where teachers reported low participation in professional learning 
activities, there were also reports of high levels of ICT use and self-efficacy, while some of 
those with high levels of participation were characterized by relatively lower levels of ICT use 


and self-efficacy. However, 


, this finding is consistent with an interpretation that suggests that in 


countries with low levels of ICT use more professional learning is offered so as to support and 
encourage further ICT use. In countries where there are already high levels of ICT use there may 
be less perceived need to provide professional learning to support its use. 


Teacher emphasis on learning CIL and CT 


Teachers’ emphasis on developing ClL-related skills 


While teachers’ self-efficacy, their perceptions of, and their attitudes toward the use of ICT 
have been highlighted as important factors influencing the implementation of digital learning at 
school (Lawrence and Tar 2018; Tondeur et al. 2017), it is also important to consider the extent 
to which teachers place emphasis on the teaching of digital skills during class (Siddig et al. 2016). 
Results from ICILS 2013 showed that across participating countries there were relatively high 
levels of teachers who reported to emphasize ClL-related skills in their teaching, and that the 


extent of this emphasis wa 


s related to their sense of ICT self-efficacy, views of the use of ICT in 


class, and perceptions of teacher collaboration regarding ICT use at school (Fraillon et al. 2014). 


To capture the emphasis on developing CIL/CT, ICILS 2018 asked teachers to answer a question 
with regard to the specific reference class at the target grade, which they had identified prior 
to the question (see introduction to this chapter). 


Teachers were asked about the emphasis they gave in their reference class to developing nine 
different |CT-based capabilities in their students (“strong emphasis,’ “some emphasis,” “little 


emphasis,’ or “no emphasis 


(a) toaccess information 
strong emphasis) 


c) toevaluate the credib 


d) toshare digi 


f) to provide digital feed 


». 


efficiently (85% on average across countries reported some or 


b) to display information for a given audience/purpose (78%) 


ility of digital information (74%) 


tal information with others (71%) 


e) touse computer software to construct digital work products (e.g., presentations, 
documents, images, and diagrams) (76%) 


back on the work of others (such as classmates) (49%) 


g) toexplore arange of digital resources when searching for information (75%) 


( 
( 
(h) to provide references 
( 


for digital information sources (67%) 


i) _tounderstand the consequences of making information publicly available online (67%). 
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Across countries most teachers reported some or strong emphasis in their teaching on these 
capabilities. All nine items were used to derive an IRT scale reflecting teachers’ emphasis on 
developing ICT-based capabilities in class. The scale has satisfactory reliabilities across countries 
Cronbach's alpha = 0.90) and items were scored so that higher scale scores indicated stronger 
emphasis on the development of ICT-related capabilities (see the corresponding item map in 
Figure F.22 in Appendix F). 


The highest levels of teacher emphasis were recorded in Italy and Kazakhstan, while teachers 
in Finland and, in particular, in North Rhine-Westphalia (Germany) reported lower levels of 
emphasis (Table 6.16). When comparing teacher emphasis across different subject areas,** not 
surprisingly the highest average scores (across countries) were reported by teachers referring 
to ICT-related classes (i.e., subjects that focus on teaching ICT-related skills). We also observed 
relatively high scores with regard to test language and human sciences classes. The lowest scores 
of teacher emphasis were observed for mathematics and classes related to other subjects such 
as vocational, moral/ethics, and physical education. These findings are broadly consistent with 
those from ICILS 2013 (Fraillon et al. 2014). 


To assess the associations of teacher emphasis with other |CT-related perceptions, we estimated 
inear multiple regression models to explain variance in teacher emphasis on teaching ICT-based 
skills. We used the following scales as predictors in this model: teachers’ self-efficacy, teachers’ 
positive views of using ICT for teaching and learning, teachers’ reports on collaboration between 
teachers when using ICT, and teachers’ reports on the availability of ICT resources at school. 
Furthermore, we included teachers’ reported experience with the use of ICT during lessons as 


an additional predictor variable. 


All predictor variables were nationally standardized to having national average scores of zero 
and standard deviations of one. Therefore, the unstandardized regression coefficients (Table 
6.17) indicate changes in teacher emphasis scores corresponding to an increase in one national 
standard deviation in each of the predictor variables after controlling for the effects of all other 
variables in the model. For teachers’ experience with pedagogical use of ICT, we coded the 
responses so that regression coefficients indicate the change in scale scores with (approximately) 
one year of further experience.?° 


Both teachers’ ICT self-efficacy and teachers’ reports on ICT-related collaboration between 
teachers were statistically significant and consistent predictors of teacher emphasis in developing 
ClLin all countries. With one national standard deviation on the ICT self-efficacy scale, there was 
(on average across participating countries) an associated increase of over two score points in 
teacher emphasis in developing CIL (equivalent to approximately one fifth of astandard deviation). 
Anincrease of one national standard deviation in the teacher collaboration scale was (on average 
associated with more than two score points increase. 


Teachers’ positive views of the use of ICT for teaching and learning were also positively and 
significantly associated with their emphasis on students’ learning of ICT in all countries; on 
average, one national standard deviation was associated with an increase of almost two score 
points. The scale scores reflecting teachers’ reports of the availability of ICT resources at schoo 
were not consistently associated with teacher emphasis on developing CIL; we found statistically 
significant coefficients only in Chile and Kazakhstan. Teachers’ experience with the use of ICT 
during lessons was consistently positively associated with their emphasis on learning ClL-related 
skills. On average across countries, one more year of experience was approximately associated 
with an increase of about one score point. 


24 Insome countries, the number of teachers in subject-area subgroups were too small (below 30) to report estimates. 
25 The response categories were coded as follows: “never” = O, “less than two years” = 1, “between two and five years” 
= 3, and “more than five years” = 5. 
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On average across participating countries, the regression model explained 27 percent of the 


variance in teachers’ emphasis scores, ranging from 16 percent in 


Denmark to 35 percent in Italy. 


However, readers should be cautious about drawing any conclusions about causality from these 


analyses, as the cross-sectional design does not allow any causal 
is possible that perceptions of teacher collaborations are also in 


interpretation. For example, it 


fluenced by the teachers’ own 


emphasis on teaching |CT-related skills, which leads to the need for more communication on 


how to achieve this. 


Teachers’ emphasis on developing CT-related skills 


In recognition of the optional assessment of CT as part of ICILS 2018, the teacher questionnaire 
also included a question about how much emphasis teachers placed on teaching the following 


CT-related skills (“strong emphasis,’ “some emphasis,” “little emph 


asis,” or “no emphasis”): 


(a) todisplay information in different ways (on average across countries 84% placed some or 


strong emphasis on this skill) 


b) to break a complex process into smaller parts (77%) 


i) touse real-world data to review and revise solutions to prob 


All nine items were used to derive an IRT scale reflecting teac 


c) tounderstand diagrams that describe or show real-world problems (65%) 
d) to plan tasks by setting out the steps needed to complete them (76%) 

e) touse tools making diagrams that help solve problems (48%) 

f) to use simulations to help understand or solve real-world problems (48%) 
g) tomake flow diagrams to show the different parts of a process (36%) 


h) torecord and evaluate data to understand and solve a problem (62%) 


lems (64%). 


hers’ emphasis on teaching CT- 


related skills in class. The scale has satisfactory reliabilities across countries (Cronbach's alpha 
= 0.90) and items were scored so that higher scale scores indicated stronger emphasis on the 
teaching of CT-related tasks (see Figure F.23 in Appendix F, for the corresponding item map, 
which illustrates how scale scores are related to expected item responses). 


The highest average scores for teacher emphasis on teaching CT-related skills were observed 
in Italy, Kazakhstan, and Moscow (Russian Federation), while we observed notably lower scale 
scores in Finland and North Rhine-Westphalia (Germany) (Table 6.18). When comparing teacher 
emphasis on the teaching of CT-related skills across subject areas, the highest scale scores were 
recorded for I|CT-related classes, followed by mathematics and natural sciences, while the lowest 


We also regressed scale scores reflecting teacher emphasis on 


scale scores were found for other language classes and other subjects’ classes. 


teaching CT-related tasks on 


(nationally standardized) scale scores reflecting teachers’ ICT self-efficacy, positive views on 
using ICT in teaching, perceptions of collaboration with other teachers for using ICT in teaching, 
views of the availability of ICT resources at school, and teachers’ experience with using ICT 
during lessons (Table 6.19). The results show broadly similar results to those we observed for 


teacher emphasis on ClL-related skills. 


Teachers’ perceptions of collaboration with colleagues regarding the use of ICT for teaching 


were consistently positive and sign 


ficantly related to teacher emphasis on CT-related skills. On 


average, an increase in one national standard deviation in the scale reflecting perceptions of 
teacher collaboration when using ICT was associated with an increase of over two scale score 


points in teacher emphasis on the teaching of CT-related tasks. Teachers’ 


CT self-efficacy was 


also consistently and significantly related to teacher emphasis on teaching CT-related skills (an 
increase of one national standard deviation corresponded to an increase of more than one 


score point in teacher emphasis) and teachers’ positive views of | 
consistent and significant predictor (with an average effect of 1.1 


CT use in teaching were also a 
scale score points). 
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Teacher reports on the availability of |CT resources at school were positive predictors only in Chile 
and Kazakhstan, a result that is similar to the multiple regression results for teacher emphasis 
on ClLrelated skills. Teachers’ experience with the pedagogical use of ICT was a less consistent 
predictor in this model. We found significant effects only in about half the participating countries 
and, on average, approximately one year of further experience was associated with about half 
ascore point of emphasis on the learning of CT-related skills. 


The regression model explained 16 percent of the variance in scale scores reflecting teacher 
emphasis on CT-related skills, ranging from nine percent in North Rhine-Westphalia (Germany) 
and 11 percent in Portugal to 29 percent in Kazakhstan. When interpreting these results, it is 
important to remember the caveats with regard to drawing causal conclusions based on cross- 
sectional data made earlier in this chapter. 


Teachers’ use of ICT for teaching and learning 


Results from previous cross-national studies such as SITES 2006 (Law et al. 2008) and ICILS 
2018 (Fraillon et al. 2014) have shown that schools and classrooms vary in the extent to which 
educators use ICT in teaching. Even though e-learning technologies are widely perceived as 
bringing transformative effects to classrooms (see Aparicio et al. 2016; Burbules 2007), their 
implementation has been relatively limited (see Cuban 2001; Stosic 2015; Vrasidas 2015) and the 
effectiveness of ICT for promoting learning appears to depend on teachers’ actual practices and 
their ability of integrate digital technologies into teaching practice (Comi et al. 2017). The ICILS 
2013 results showed that, while majorities of teachers reported using ICT for teaching, they used 
it more frequently for relatively simple tasks than more complex tasks (see Fraillon et al. 2014). 


Teachers’ use of ICT tools in class 


CILS 2018 asked teachers to rate the frequency (“never,’ “in some lessons,” “in most lessons,’ 
or “in every, or almost every lesson”) with which they used 16 different tools when teaching 
the nominated reference class during the current school year. The tools can be subdivided into 
general utility ICT tools and (more specific) digital learning tools. 


There were differences in the percentages of teachers who reported using various general utility 
CT tools “in most lessons” or “in every, or almost every lesson” (i.e., combining the two highest 
frequency categories) (Table 6.20). On average across countries, the most frequently used general 
utility tools were those for word processing (43%) and presentations (43%), followed by using 
computer-based information for capture and editing (36%), and digital contents linked with 
textbooks (82%). Meanwhile, spreadsheets (17%), video and photo software for capture and 
editing (15%), and communication software (22%) were less frequently used by teachers. There 
were also some notable differences across countries: while teachers in Kazakhstan, Korea, and 
oscow (Russian Federation) tended to report higher levels of use across the different tools, 
the lowest levels of use were recorded in Italy and North Rhine-Westphalia (Germany). 


When reviewing the percentages of teachers who reported using digital learning tools in most 
essons, or in almost every or every lesson (Table 6.21), we found generally lower proportions of 
teachers with frequent use than for general utility software. The most frequently used learning 
tools were learning management systems, which, on average across countries, 28 percent of 
teachers reported as used in most lessons or more frequently. In this context it is worth noting 
that across participating countries, teachers expressed less confidence in using these tools 
compared to other forms of ICT use (see Table 6.2). The proportions of teachers that were 
using this type of tool varied considerably, from only five percent or less in Chile, Italy, Portugal, 
and North Rhine-Westphalia (Germany), to more than half of the teachers in Finland (53%) and 
Kazakhstan (73%). 


PREPARING FOR LIFE INA DIGITAL WORLD 


208 


as8esJaAe MOjaq SjUIOd a8e]UIIINd OT UY} 2IO\| A 


adesane Mojaq AjjuedYyIUBIS A, 


“JedA |OOUDS dU JO J/eY JSuY 9} Ul apes Jase} paAdanins AyyuNOD 
‘uolze/Ndod jase} |eEUOIJEU BU} JO %GS 0} %OS SJ9A09 uoHjejndod pauyap |euolzeN 


z 


T 


asesane aroge AjjuedYyIUZIS V7 “papnjoul d4aM sjooyps JUaWAaTe| das Jaye AjUO Sayed UOIedDIqed Buljduues OJ sauljapIns jay), 
aseiaAe adoge sjulod a8ejuadsad OT ueyy oso) Ww ‘squawauinbad uoledioiqied ajduies 8uljsaw sjuedio qed 8ulysewiyouag JO Sal4jzuNnod 404 paysoda A|UO BTOZ% STD YM suOsIeduo7 
sade SINSB4I 8TOZ STD] |euoeN "juays|suooUl Jeadde Ae $}e}0} SWOS “JSGUINU BjOUM 4S3JESU BY} O} P9PUNOI d4e SINSB asnedag ‘sasayjuaJed ul seadde sjOJJa PsepueTS :S9JON 
A (50) S A (60) 2 A (90) S A (90) v A (S50) € A (OT) TL A (60) 27 (AuetuJad) eljeydysa/y\-auiyy Yo 
Vv (ST) Ev Vv (ST) TS Vv (ST) SE VV (OT) 8t Vv (vt) 62 Vv (8t) 729 Vv (8T) 6S (uolJeJapa_ UeISsNY) MOISO 
sjuawauinbad uoljzedioijied ajdwes 3unjaauw sjzuedioijied 3ulyseuysueg 
(OT) 9t (ET) 82 (ET) ST (60) TT (80) TT (TT) 81 (ET) vz Aensnin) 
(ET) €% (QT) 8E (ST) SE (OT) 9T (OT) vt (VT) Sv (97) Sv pS9Fe}S pou 
(vt) TT (07) 02 (ST) ST (ST) 8 (VT) 8 (TZ) TE (LT) €€ BINOquIaXN] 
(£0) 9 (ET) ET (TT) Z (LO) 7 (L0) 9 (VT) 87 (97) Te Auewtad) 
(CT) (LT) 81 (60) OT (OT) @ (LT) OT (ST) 9€ (LT) 9S pooues4 
sjudWaJINbas uoled!aij4ed ajduwies sayde9} 8U1JI9W JON 
(90) ZE (90) 9E (SO) 7% (SO) ST (SO) ZT (90) €v (LO) €v asesaAe BLOT SIDI 
Vv (TT) 8v (VT) ve A (OT) 02 A (£0) Tt A (£0) vt Vv (@T) €s A (vt) 9€ |esnyod 
Vv (ZT) @ A (v'T) O€ (VT) €Z Vi (€Z) 9% VY (8T) cz vi (€?) vs Vi (92) 6P Jo aIIGNday ‘eatoy 
(TZ) TE V (07) 9v Vv (ST) Sv Vv (2?) SE Vv (6T) 97 v (Oo?) 29 v (St) 29 sUEISUAEZE 
(ST) ve (9T) €€ A (80) OT A (80) 8 A (90) S A (ZT) 02 A (VT) €% Ale} 
(ET) CE (VT) LE V (ET) SZ A (50) S A (S50) v A (OT) 82 A (OT) Zé pue|ul+ 
A (ET) 8T V (ST) 6€ A (ET) 2 A (£0) S A (TT) St A (6T) €€ Vv (02) v9 , PHewuaq 
A (ET) OT A (ET) TE (ET) OZ A (TT) €t A (ST) €T V (St) Lv A (LT) 8€ a!IUD 
([doysojoyd aqopy 
(elpaedojaA.ua ‘SIAOP|! JAE 
‘SDM ‘Saqisqam (adAys alAO|W SMOpUl A] “3") ([@ WUIog4amog ([@ PAom 
$00q}x9} paye|as-d1d0} “3'a) ‘Bulsessaw aJIp 3uljIpa pue ([@ |82xF YOSOIIIN|] YOSOIII\] 
YUM paxul| $991NOSAJ UO!}EWJOJU! “lea “3"a) a4eMIOS ainjded 10} a14eMyos JYOSOJDI\\] “3"8) “3'9) a4eMyos “3'8) a4eMYos 
$]U9}UOD |e}IZIG paseq-iajyndwo5 uoljed1uNWWO> ojoud pue oapi/), sjaayspeaids uol}e]UaSa1d JOssaz01d-pso/\, 
:J0 asn JuaNba.y 3ulj10daJ siayded} JO 98e}U9I19g Aayunod 


uossa| AlaAa JO ‘AJBA9 JSOLUD ‘SUOSSA| ISO U! $}00} | D] ARI IIZN JOJaUas Suisn payiodas OyM s1ayove} Jo Sa80}UID/Ad JOUOIIDN ‘O79 a|qoL 


209 


“JeaA |OOYIS dU} JO FEY JSAY BY} Ul ApeIB yaBe} padaruns AsjuUNOD 

‘uol}e|NdOd JaB.1e} |eUO!ZEU 3} JO %GS 01 % OS S4BA09 uOlJejNdod pauyap jeuojen , 

‘papn|oul 349M sjooups JUdWAIe|daI Jaye AjUO Sazes UO!JeEdIoIVJed Buljdwes JOJSaUljapINs apy, 
‘squawasinbad Uo!ed|d1J4ed ajdwes 


aseiaAe MOjaq squlod asejUadJad OT UEY} QO) A z 
aseuaAe Mo|aq AjJUEDYIUBIS A 


asesane aroge AjjUedYyIUZIS V7 


adesane anoge sjulod a8ejualJed QT ueYy} ao Ww SUlJaaW SjUed|dZued BUlyJeLUYUDG JO SA11}UNOD JOJ paysoda. AjUO BT OZ STD YM SUOSIedWOZ jUa}sISUODU! Jeadde Aew 
d4E S}INSBI QT OT STD |EUOEN S]€]O} BWOS VAquUNU ajOYM 4sasedU BY} O} PapuNo ase S}jNsa asnedag ‘sasayjuaed ul Jeadde siossa psepueys :S9JON 
A (ZO) T A (€0) T A (S0) 7 A (SOE A (70) Z A (v0) 2 A (€0) Tt A (20) T A (£0) T A (€0)T (Aueuad) eljeyd}sa/-aulyy 4340 
(CT) ST (£0) 9 V (tT) €t VY (vt) O€ V (eT) 02 A (01) 6 (LO) S (8'0) A (80) OT (OT) 6 (UoljeJapa  Uelssny) MOISO 
sjuawasinbas uoedioiqed ajdwes 8uljaaw s}uedioiqed 8upjsewysueg 
(80) 9 (VT) et (OT) TT (OT) &T (ZT) €T (TT) 21 (90) € (£0) 9 (60) Z (80) 9 Aensnin 
(90) Z (Z0) 9 (60) TT (CT) SZ (LT) vv (ST) Le (£0) 8 (£0) Z (ET) 6T (LT) 2 1597235 Poy 
(80) € (SO) T (ET) 9 (ET) Z (9T) C1 (80) € (v0) T (£0) T (OT) v (ZTE S.noqwaxny 
(EO) Z (EO) T (90) S (vO) € ($0) @ (SO) (v0) Z (EO) T (€°0) Z (v'0) Z AUBW9S) 
(€0) T (0) T (90) S (£0) 9 (90) 9 (0) T (SO) Z (SO) Z (SO) v (vO)¢ pOOUesy 
sjuawainbas uoljedioijied ajdwes Jaysea} 8Ulj99W JON 
(SO) 21 (v0) Z (£0) OT (SO) 2 (SO) ZT (SO) 82 (E70) S (€0) Z (v0) 2 (v0) TT asesane 8TOZ SIDI 
A (S50) v A (70) Z A (€0) 7 A (80) ST A (£0) TT A (50)S A (v0) 2 A (€0) Z A (£0) 6 A (FO) L jesnyod 
V (8ST) 1 Vi (VT) 6 VY (80) TT A (80) 27 A (TT) €T (ST) 92 (ET) 9 (ET) 8 (£0) 1 VY (TT) Ot Jo dI]Gnday ‘ea 
Vv (22) 8€ Vv (8t)t@ | W (ST) OF | W (07) Or Vv (67) 6€ Vv (Vz) eZ v (LT) 8t Vv (ST) 6t Vv (27) 9 V (ST) OZ jUEISUAEZE| 
A (SO) € A (20) 1 A (v0) S A (60)6 A (S0)S A (Z£0)S A (€0) t YV_ (60) OF A (90) S A (90)S Aled 
A (90) S A (90) ¥ A (€0) € A (S0)9 A (OT) €T Vv (vt) es A (v0) Z A (€0) T A (90) 9 V_ (80) €T pue|ul 
A (60) S A (€0) T A (£0) Z Vv (12) 67 Vv (27) 62 V (22) 2 A (SO) € A (S0) Z A (80) S (OT)6 1 Peed 
(ET) 21 V(t) tt A (ZO) 8 (CT) TZ A (80) 6 A (S50) € A (50) € (£0) Z (VT) TT A (60)S a!YD 
([4ossajyzew] 
‘ooyey, 
‘421ZIND] “3'2) 
suol}sanb 
(sjoafqo (21Ped] ([p4aeoqy5e|q] ((Quoesdsqay] syuapnys 
Sulued| ([eyouUaUC] [@ ‘[opow py] ([030739N] ‘[@ uorjestdsuy] se NoA 
([42}IML ([peesy,ea10A] eEMYOS “3'a) Se4n0se. s90q 9/3005] “3'9) Wayshs “3'9) a4eMyos “3'9) auemMyos asym sdde 
‘yooqaze4] “3'a) “3'3) 3uIMeJp JO Bulusea] jeysip | “8*e) a4emyos judwaseuew 3uljapouw pue suiddew sowes 8ulused| Jo sweiz01d 
e1pau [2190S SOl|ojjsod-a Bulydeiy 9AI}DC19}U] aalzesoge||OD Bulusea] 7 suol}e|NWIS jdaou0D Jeusiq 901}981d 
1JO asn Juanba. 8ulzJ0da sidyrea} JO a8e}UII19q Aayunoz 


TEACHING WITH AND ABOUT ICT 


Uossa] AJAAd JO A1BA9 JSOUI|D ‘SUOSSA] JSOL Ul $/00} | 9} SUlUa| [D}SIp Suisn paziodas oyM sayova} Jo Sa80}UIDIAd JOUOHON :L.Z'9 alqv]. 


210 


PREPARING FOR LIFE INA DIGITAL WORLD 


nteractive learning resources were more frequently used 
tools. In addition, there were notable between-country va 
who reported more frequent use of interactive learning 


Denmark. Frequent use of collaborative software was re 
on average, again with considerable variation in the propo 


(22%) than most other digital learning 
riations in the proportions of teachers 
resources, ranging from less than 10 


percent in Chile, Italy, and North Rhine-Westphalia (Germany), to almost half of the teachers in 


ported by 17 percent of the teachers 
rtions across countries. 


ore frequent use of practice programs or apps to ask students questions, digital learning games, 


graphing or drawing software, and social media were, on average, reported by about 10 percent 
of the surveyed teachers, again with some variations across countries. The least commonly used 
digital learning tools (reported by on average < 10% of teachers) were concept mapping software, 


simulation and modeling software, and e-portfolios. 


Teachers’ use of ICT for class activities and teaching practices 


their students often or always used ICT; data from teach 


had not engaged in each of these activities were not included in these percentages (Table 6.22). 


CILS 2018 was also interested in finding out about the extent to which ICT was used for 
different types of learning activities in class. Teachers were asked to provide information about 
whether their students engaged in a set range of activities, and how often they used ICT as 
part of these activities (“they do not engage in this activity; “they never use ICT in this activity,’ 
“they sometimes use ICT in this activity,’ “they often use ICT in this activity,’ or “they always 
use ICT in this activity”). We present the results as percentages of teachers who reported that 


ers who indicated that their students 


On average across participating countries, half or almost half of the teachers reported that 
students frequently used ICT when collecting data for a project (50%), creating visual products 
or videos (47%), working on extended projects of more than a week (46%), and working on short 
assignments of less than a week’s duration (46%). More than a third of teachers (on average) also 


information resulting from a search (3 
of ICT by students when analyzing data (32%), communic 
in other schools (28%), explaining and discussing ideas w 


for themselves (22%). 


We also observed notable differences across countries. 


CT for these different learning activities, they tended to be 
and Moscow (Russian Federation). When reviewing these 
these percentages are based on only those teachers who 


on extended projects, to 31% for reflections on learning ex 


reported that students often or always used ICT when submitting completed work assessments 
41%), sharing products with other students (40%), working individually on learning materials at 
their own pace (37%), undertaking open-ended investigations or fleld work (35%), and evaluating 
1%). Less than a third of teachers reported frequent use 


ating on projects with other students 
ith other students (26%), engaging in 


reflections on their learning experiences (22%), and planning a sequence of learning activities 


While relatively low proportions of 


teachers in Finland, Portugal, and North Rhine-Westphalia (Germany) reported frequent use of 


much higher in Denmark, Kazakhstan, 
results, readers should consider that 
reported that their students engaged 


in each of the learning activities, and that there were substantial proportions of teachers who 
reported that these were not undertaken in their reference class (ranging from 10% for work 


periences). 


The ICILS 2018 teacher questionnaire also included a question regarding their use of teaching 


practices and the frequency of use of ICT when applying 


them (“I do not use this practice with 


the reference class,’ “| never use ICT with this practice,’ “| sometimes use ICT with this practice? 
“| often use ICT with this practice.’ or “I always use ICT with this practice”). Again, the results 


are presented as percentages of teachers who often or a 


ways used ICT based on all teachers 


who reported using each of these practices (teachers who did not use this practice with the 


reference class were excluded from the calculations). 
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On average across ICILS 2018 countries, almost two-thirds of teachers who presented information 
through direct class instruction reported often or always using ICT (64%) (Table 6.23). Between 
a third and a half of those teachers often or always made use of ICT when communicating with 
parents or guardians about students’ learning (45%), when supporting student-led whole-class 
group discussions and presentations (43%), when providing remedial or enrichment support 


to individuals or groups of studen 


ts (40%), when supporting inquiry learning (40%), and when 


assessing students’ learning through tests (38%). Less than a third of teachers reported more 


frequent use of ICT when providin 


g feedback to students on their work (32%), when supporting 


collaboration among students (31%), and when mediating communication between students 


and experts or external mentors (26%). 


Again we observed considerable variation across countries. While teachers in Denmark, 


across the different teaching prac 
orth Rhine-Westphalia (German 
the percentages were based only 
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CHAPTER 7: 


Investigating variations in computer and 
information literacy and computational 
thinking 


Chapter highlights 


There are differences in the proportion of variance between schools in computer and 
information literacy (CIL) and computational thinking (CT) scores. 


There were considerable differences in the variance for both CIL and CT as well as the 
proportion of variance found between schools across participating countries. (Table 7.1 
and Table 7.6) 


Multilevel models explained most of the variance in CIL and CT scores at the school level 
while less variance was accounted for within schools. (Table 7.1 and Table 7.2) 


There are consistent net effects on CIL and CT by personal and social background factors. 


Female gender tended to be positively related to CIL but negatively related to CT scores. 
Table 7.2 and Table 7.7) 


Use of test language was a positive predictor of CIL and CT insome countries, in particular 
those with larger proportions of students speaking another language at home. (Table 7.2 
and Table 7.7 


Both expected university education and socioeconomic background are consistent 
positive predictors of both CIL and CT across countries, however, the effect sizes vary 
across countries. (Table 7.2 and Table 7.7) 


There are consistent net effects on CIL and CT by a number of student-level predictor 
variables related to experience and use of information and communication technology (ICT) 
across countries. 


Students’ daily use of ICT and experience with computers were consistent positive 
predictors of both CIL and CT. (Table 7.3 and Table 7.8) 


Availability of computers at home was a positive predictor in most countries but had 
weaker effects after controlling for personal and social background. (Table 7.3 and Table 
7.8) 

Student reports on having learned about Cll-related tasks at school and on the use of 
general ICT applications in class tended to be a positive predictor of student CIL in a 
number of countries. (Table 7.3) 

Student reports on having been taught CT-related tasks had negative effects on CT scores 
across countries. (Table 7.8) 


There are mostly inconsistent net effects on CIL and CT by school-level predictor variables 
related to ICT across countries. 


Schools’ socioeconomic context (as measured by aggregated student scores reflecting 
family background) was aconsistent positive predictor of ClLand CT in almost all countries. 
(Table 7.2 and Table 7.6) 


In the few countries where ICT-related school-level predictor variables had significant 
associations, these were often no longer significant after controlling for social context 
variables. (Table 7.5 and Table 7.10) 
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Background 


n previous chapters we have described some of the associations of computer and information 
iteracy (CIL) and computational thinking (CT) with a number of variables, for example gender 
or those related to home background. This chapter investigates the combined influence of a 
number of variables on variations in CIL and CT including individual (student level) as well as 
context (school level) variables. This chapter addresses, for both CIL and CT, the following 
research questions: 


RQ2 What aspects of schools and education systems are related to student achievement? 


RQ3 What characteristics of students’ levels of access to, familiarity with, and self-reported 
proficiency in using computers are related to student achievement? 


RQ4 What aspects of students’ personal and social backgrounds (such as gender, socioeconomic 
background, and language background) are related to student achievement? 


n this chapter, we use multilevel models to review the extent to which different factors at the 
student and school level are associated with variations in CIL and CT scores. Factors of interest 
will be those related to access to, use of, and familiarity with information and communications 
technology (ICT), school context factors, as well as variables reflecting the personal and social 
background of students. 


Prior to the International Computer and Information Literacy Study (ICILS) 2013, research into 
earning outcomes related to ICT and factors influencing student knowledge in this area had 
generally been limited to national studies. In a number of countries these national evaluations 
provided evidence about factors explaining variation in |CT-related capabilities among students. 


Sample surveys carried out as part of the Australian National Assessment Program for ICT 
iteracy showed that students’ gender (female), socioeconomic background, and experience and 
current use of computers were positive predictors of variation in ICT literacy (ACARA [Australian 
Curriculum, Assessment and Reporting Authority] 2012, 2015; MCEECDYA [Ministerial Council 
for Education, Early Childhood Development and Youth Affairs] 2010; MCEETYA [Ministerial 
Council on Education, Employment, Training and Youth Affairs] 2007). In the United States, 
results from the National Assessment of Educational Progress Technology and Engineering 
Literacy assessment indicated substantial differences with regard to the ICT skills scale between 
gender groups, parental education, ethnic background, and school location (US Department of 
Education, National Center for Education Statistics 2016). 


CT literacy was also assessed as part of the Chilean national assessment of students’ ICT 
competencies, known as SIMCE TIC. Multilevel analyses of these data illustrated considerable 
variation among schools as well as effects of cultural background, socioeconomic status, a 
school characteristics (private/public, subsidies) on digital competencies (Roman and Murr 
2013). Further analyses also provided evidence of strong effects of prior achievement in read 
and mathematics on digital competencies (San Martin et al. 2013). 


nd 
llo 
ng 


Analyses of Norwegian grade 9 data collected in 2009 emphasized the importance of home 
factors (such as family background) but also of having a supportive school climate (Hatlevik 
2009). In their analyses of survey data from Norwegian upper-secondary schools, Hatlevik and 
Christopherson (2013) revealed substantial variation within and between schools, with home 
conditions and academic aspirations as important predictors of digital competence. Multilevel 
analyses of data collected among grade 7 students also highlighted the importance of mastery 
orientation and self-efficacy as predictors of digital competence (Hatlevik et al. 2015). 


At the international level, the Programme for International Student Assessment (PISA) assessed 
the performance of 15-year-old students in digital reading across 16 countries (OECD 
[Organisation for Economic Cooperation and Development] 2011). Eventhough this international 
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study assessed reading competencies in a digital environment, it also partly reflects skills related 
to CIL. Results from this study showed that socioeconomic background and computer use were 
positively associated with digital reading skills. However, no clear association was found with 


the use of computers at school. 


Multilevel analyses carried out using data from ICILS 2013 showed that students’ experience 
with computers as well as regular use at home had significant positive effects on CIL in many 
countries even after controlling for the influence of personal and social context. ICT resources, 


in particular the number of com 


background was taken into account (Fraillon et al. 2014). 


Data and methods 


To develop a model to explain variation in each of CIL and CT we used prior research literature 


and the contextual framework for ICILS 2018 in order to determine possib 


variation in CIL and CT for inclusion | 
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research and results from other analyses of this study 


h gender, students’ expectations of educational attainment, 


ioeconomic status are associated with students’ CIL. These variables were 


included at the student level in 
of the models used). 


Social context of schools: The average socioeconomic status of the student body was used 
as a factor that has been shown to be associated with a variety of learning outcomes. This 
variable was included at the school level in Model 1 and Model 3. 


and Model 3. 


odel 1 and Model 3 (see later for more detailed discussion 


CT resources and use at home: These predictors include ICT resources at home, persona 
experience with ICT, students’ use of ICT at home and school, and students’ experiences with 
earning about ICT at school. These variables were included at the student level in Model 2 


CT resources and use at school: Information on the schools’ ICT resources and ICT use were 
collected through the ICT coordinator, principal, and teacher questionnaires. The school CIL 
earning context includes the expectations of school principals regarding teacher use of ICT 
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The personal and social student background characteristics includ 


The schoo 


for collaboration/communication at school and teacher reports of their average experience 
of the use of ICT for teaching at school. These variables were included at the school level in 


Model 2 and Model 3. 


Students’ gender: This va 


Students’ use of the test language: This variable was coded as 1 


at home most of the tim 


riable was coded as 1 for females and O for males. 


e and O for other students. 


ed in the analysis were: 


for speaking the test language 


Students’ expected university attainment: This variable reflects students’ expectations to attain 
a university education (coded as 1 = expected, O = not expected). 


Students’ socioeconomic background: This variable is composite index standardized to have a 


mean of O and astandard deviation of 1 within each country and centered on school averages 
so that it indicates the effect of socioeconomic background within schools. The index consisted 
of factor scores from a principal component analysis of: 


— Highest parental occupation (as indicated by the international socioeconomic index of 


occupational status scores of both parents); 


- Highest parental education (categorical variable with O = lower-secondary or lower 
education, 1 = upper-secondary education, 2 = post-secondary non-university education, 
3 = university education); and 


— Number of books at home (categorical variable with O = O-10 books, 1 = 11-25 books, 2 


= 26-100 books, 3 = 


more than 100 books). 


s’ social intake was measured with the following variable: 


e School! socioeconomic context: This variable reflects the average of student scores on the 


The following variables indicated ICT resources and use at home: 


composite index of socioeconomic background. It indicates th 


background of enrolled 


Number of computers at home: Students reported the numbe 


computers, the resultin 
or only one computer). 


e€ average socio economic 


students and the resulting social context in which students learn. 


r of desktop and portable 


g indicator variable was coded 1 (two or more computers) and O (no 


Experience with computers: This variable reflects how long the individual student has used 
computers and was coded as 1 (five or more years of experience) and O (less than five years 


of experience). 


Use of ICT: This variable reflects the frequency with which each individual student uses ICT 


at school or outside of school for 


coded as 1 (daily use) and O (less than daily use). 


Students’ reports on using general ICT applications in class: The three-item scale is b 
a question that required students to indicate with what frequency they have use 
processing software, presentation software, and computer-based information sources at 
school. Values are item response theory (IRT) scores, which were standardized for this 
within each country to having a mean of O and a standard deviation of 1. 


Students’ reports on learning of CIL or CT: This consisted of 


— Students’ reports on 
eight items that requ 
tasks*° at school. Va 


learning of ClL-related tasks at school: The index is based on 
ired students to indicate whether they had learned about diffe 
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26 The ClLrelated tasks were: 
information for a given audience or purpose using ICT; work out whether to trust information from the internet; 


decide what information ob 
obtained from internet sourc 
use ICT to collaborate with o 


ained from the internet is relevant to include in school work; organize in 


provide references to internet sources; search for information using ICT; 
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each country to havi 
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tudents’ reports on 


The following school-level 


During multivariate analyses, any issues relating to missing data tend to become more prevalent 
ms of analysis because of the simultaneous inclusion of numerous variables. 
address the missing data issue, we first excluded from the analyses the small proportion of 
students for whom there were no student questionnaire data (this was viable because only sma 
proportions of students had missing data for the student-level variables). For the variables derived 


than in other for 


To 


fro 
(ex 


Availability of ICT resourc 
questionnaire, was com 


nine items that requi 
CIL tasks?” during the current school year. Values are IRT scores, which were standardized 
for this analysis withi 
This variable was only included in the model explaining variation of CT. 
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ng amean of O and a standard deviation of 1. This variable was only 
explaining variation of CIL. 


learning of CT-related tasks at school: The index is based on a set of 
red students to indicate whether they had learned about different 


n each country to having a mean of O and a standard deviation of 1. 


predictors reflect ICT resources and use at school: 


es for teaching and learning: This measure, based on the ICT coordinator 
puted using reports on the availability of 13 different computer and 


ICT resources.”8 The items were coded as available to teachers and students (2), available to 
either students or teachers (1), and not available (0), so that higher IRT scale scores indicate 
more ICT resources at school. Values are IRT scores, which were standardized for this analysis 
at the student level within each country to having a mean of O and a standard deviation of 1. 


School principals’ reports on expectations for teacher communication using ICT: Principals reported 
whether teachers at the school were expected and required, expected but not required, or 


deviation of 1. 


Average time of teachers’ 


not expected to collaborate or communicate via ICT with teachers, parents, and students. 
The three items were used to derive an index where higher scale scores represent higher 
expectations/requirements in this respect. Values are IRT scores, which were standardized for 
this analysis at the student level within each country to having a mean of O and a standard 


experience with using ICT for teaching: This index reflects the average 


time teachers at the school reported to have used ICT for teaching purposes. Response 


categories for the item were scored as O (never), 1 (less than two years), 3 (two to five years), 


and 5 (more than five ye 


ars). 


Teacher reports on ICT use for class activities: This IRT scale is based on a question asking 
teachers about the extent to which students used ICT for activities in class, and higher scores 


reflect greater use of IC 


the student level within each country to having a mean of O and a standard deviation of 1. 


mthe ICT coordinator q 


pectations for teacher collaboration/communication via ICT), and the teacher survey (teacher 


T. Values are IRT scores, which were standardized for this analysis at 


uestionnaire (ICT resources at school), school principal questionnaire 


experience with ICT use during lessons, ICT use for class activities) there were higher proportions 


of 


missing data. These were treated by setting missing values to national averages or modes 


27 The CT-related tasks were: to display information in different ways; to break a complex process into smaller parts; 


complete them; to use too 


o understand diagrams that d 


solve real-world problems; to 


data to understand and solve a problem; and to use real-world data to review and revise solutions to problems. 
28 The following ICT resources were used for scaling: digital learning resources that can only be used online; access to 
the internet through the school network; access to an education site or network maintained by education authorities; 


email accounts for school-r' 


capture and editing (e.g., [Wi 


[Inspiration ®], [Webspiration ®]); data logging and monitoring tools (e.g., [Logger Pro]) that capture real-world data 


s to make diagrams that help solve problems ; to use simulations to help understand or 


elated use; practice programs or [apps] where teachers decide which questions are asked 
of students (e.g., [Quizlet, Kahoot], [mathfessor]); single user digital learning games (e.g., [languages online]); multi- 
user digital learning games with graphics and inquiry tasks (e.g., [Quest Atlantis]); video and photo software for 


escribe or show real-world problems; to plan tasks by setting out the steps needed to 


make flow diagrams to show the different parts of a process; to record and evaluate 


, 


ndows Movie Maker, iMovie, Adobe Photoshop]); concept mapping software (e.g., 


digitally for analysis (e.g., speed, temperature); a learning management system (e.g., [Edmodo], [Blackboard]); graphing 
or drawing software; e-portfolios (e.g., [VoiceThread]); digital contents linked with textbooks. 
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respectively, and then by adding a missing indicator variable for missing school data (for each of 
the two indicators) and another one for entirely missing teacher data. This particular approach 
(see Cohen and Cohen 1975) was chosen given its simplicity and the relatively limited amount 
of missing values in a majority of countries. 


On average across participating countries, data from about 92 percent of tested students were 
included inthe analysis. Intwo countries (Germany and Uruguay) and one benchmarking participant 
North Rhine-Westphalia, Germany) this proportion was below 85 percent and the results are 
annotated accordingly. For these countries, as well as those that did not meet the IEA sampling 
participation requirements, readers should interpret results with due caution. More detailed 
information on the multilevel modeling and treatment of missing data will be presented in the 
CILS 2018 technical report (Fraillon et al. 2020). 


n Luxembourg, student sample participation requirements were met and 38 of a total of 41 
schools participated in the ICILS 2018 survey. Given this relatively low number of units available 
for analyses at the school level, which would have led to reduced statistical power and precision at 
the school level, data from this country are not included in the analyses presented in this chapter. 


Due to the hierarchical nature of the survey data, we conducted multivariate multilevel regression 
analysis (see Raudenbush and Bryk 2002). We estimated, for each national sample, two-level 
hierarchical models with students nested within schools. The analyses were carried out using 
the software package Mplus (version 7; see Muthén and Muthén 1998-2012) and estimates 
were obtained after applying appropriate sampling weights at the student and school level. 
The school-level weights reflect the probabilities of schools to be selected as well as potential 
non-response adjustments, while student-level weights reflect the probabilities of individual 
students to be sampled within schools as well as possible adjustments for non-response. The 
weights were scaled so that the sum of weights is equal to the number of units at each level. 


When interpreting results from a multilevel analysis, it is also important to be aware that student 
level variables in a two-level model have a different meaning from those ina single-level regression 
analysis. This is because student-level coefficients reflect the effect a variable has within schools. 
Because of this, effects at this level may differ from the findings that emerged from the bivariate 
analyses reported in previous chapters.” 


In addition to estimates of overall and explained variance at the two levels in each of the 
countries, we will present unstandardized regression coefficients with indications of their statistical 
significance (p < 0.05). These predictor variables were coded in ways that allow a substantial 
interpretation of these coefficients. Positive coefficients indicate that the estimated net increase 
in CIL or CT score points equivalent with a corresponding increase of one in the predictor 
variable, while negative coefficients estimate a decrease. With regard to the interpretation of 
what a value of one means for each factor, we can distinguish three types of predictor variables: 


e For dichotomous indicators, the regression coefficient reflects the estimated net effect in CIL 

or CT score points between the group with code of one when compared to all other students 
(female versus male students, speaking test language at home versus others, expecting a 
niversity qualification versus others, having two or more computers at home versus others, 
aving at least five years of experience with computers versus others, daily ICT use versus 
thers).°° 


Oo > € 


29 Multilevel analysis allows the estimation of random effects models, where within-school effects vary across schools 
as well as interaction effects between school-level predictors and the slopes of student-level predictors within 
schools. However, in this first analysis of ICILS data regarding factors influencing CIL, all student-level effects were 
estimated as fixed effects that did not vary across schools. 

30 For example, a coefficient of 5 for the gender indicator denotes that (within schools) females are predicted to have 
five score points more than males after controlling for all other variables, while a coefficient of -5 would indicate that 
males are expected to have five score points more than females. 
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Model 1 (including the personal and social context factors) explained, on average, eight percent 
at the student level (ranging from 3% in Germany to 16% in France) and 48 percent at the school 
level (ranging from 13% in Kazakhstan to 79% in North Rhine-Westphalia, Germany). Model 2 
(including |CT-related factors) explained on average 10 percent at the student level (ranging from 
6% in Portugal and North Rhine-Westphalia, Germany, to 17% in Finland) and 37 percent at the 
school level (ranging from 25% in Denmark and Kazakhstan to 50% in Finland and Germany). 
The final Model 3 (including all predictors) had an average variance explanation of 16 percent 
at the student level (lowest with 11% in Kazakhstan and highest with 24% in Finland) and 64 
percent at the school level (lowest with 33% in Kazakhstan and highest with 86% in Uruguay). 
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n all participating countries (with the exception of Germany), expected university education 
was strongly associated with CIL, on average the effect was 27 score points in Model 1 and 23 
in Model 3, with considerable differences across countries. Larger differences across countries 
were observed for the effect of socioeconomic background, both at the student level (on average 
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Students’ reports on the use of general ICT applications during class had positive associations with 
ClLin five out of 10 countries, and on average one national standard deviation was associated with 
over five points in Model 2 and less than five points in Model 3. Students’ perceptions of having 
learned about ClL-related tasks was a significant predictor in five countries and the benchmarking 
participant Moscow (Russian Federation) in Model 2 (with an average effect of almost four points), 
and in four countries in Model 3 (on average associated with less than three points). 


When reviewing the effects of school-level predictors, we observe less consistency in the 
associations of these variables with CIL (Table 7.4). School expectations of teacher collaboration 
or communication via ICT was a significant positive Model 2 predictor in Germany (with 23 
points) and Portugal (with 12 points), as well as in the benchmarking participant North Rhine- 
Westphalia (Germany). In Model 3 this factor was only significant in Germany (with a regression 
coefficient of 10 points). 


CT coordinator reports on the availability of school resources had significant positive effects on 
CIL in Germany (24 points) for Model 2. This variable was a positive predictor in Germany and 
Portugal for Model 3 after controlling for social context variables. 


The average years of experience of teachers using ICT for teaching was a significant positive 
odel 2 predictor of CIL in Chile, Kazakhstan, Uruguay, and the benchmarking participant of 
orth Rhine-Westphalia (Germany). However, after controlling for personal and social background 
variables in Model 3 this was only a significant positive predictor in Chile and North Rhine- 
Westphalia (Germany). Teacher reports on students’ ICT use for class activities had significant 
positive associations with CIL for Model 2 in Finland and France. However, no significant effects 
in any of the countries meeting IEA sample participation requirements were recorded after 
controlling for social context variables as part of Model 3. 


We summarized the results by displaying the numbers of significant positive and negative effects 
per country (by accounting only for countries meeting IEA sample participation requirements) 
in each of the models (Table 7.5). For example, for (female) gender, we recorded six instances 
where this Model 1 predictor had statistically significant positive effects across countries, while 
there were no countries where this variable had a negative effect. The same numbers were found 
when including all predictors in Model 3. Generally, we can observe that while student-level 
|CT-related variables remain significant after controlling for personal and social context variables 
(i.e., comparing Model 2 and 3), the few significant associations of school-level variables in Model 
2 are further reduced after controlling for social context factors in Model 3. 


Explaining variation in CT 

To explain variation in students’ CT scores, we applied almost the same models as for CIL. The 
only difference was the use of the scale reflecting students’ report on learning about CT-related 
tasks, instead of including the scale reflecting students’ perceptions of having learned about Cl 
related tasks at school. The analyses included data from six countries that met the IEA sampling 
participation requirements, data from the United States (which were reported separately as they 
had insufficient participation rates), and data from the benchmarking participant North Rhine- 
Westphalia (Germany) (data which are also included in the German national sample). 


The results regarding the variance estimates for CT (overall, within, and between schools) and 
variance explanation by different models at each level are somewhat similar to those for CIL. 
(Table 7.6). There was considerable variation with regard to the overall variance of CT scores 
across countries, ranging from slightly below 7000 in Denmark and Portugal to more than 12,000 
in Germany and Korea. The proportion of variance between schools also differed considerably, 
ranging from seven percent in Denmark to 46 percent in Germany. On average across countries, 
the proportion of variance between schools was 20 percent. Model 1 predictors (related to 
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personal and social background) explained on average eight percent within schools (ranging 
from 3% in Germany to 14% in France) and 43 percent between schools (ranging from 18% in 
Korea to 79% in Germany). The corresponding estimates of explained variance for Model 2 were 
nine percent within schools (ranging from 5% or less in France, Germany, and the benchmarking 
participant North Rhine-Westphalia, Germany, to 15% in Finland) and 33 percent between schools 
ranging from 16% in Korea to 46% in Finland). For Model 3 these were 15 percent on average 
within schools (lowest was 8% in Germany and highest was 19% in Finland) and 58 percent at 
the school level (lowest was 26% in Korea and 86% in Germany). 


When reviewing the unstandardized regression coefficients for variables related to personal 
and social background (Table 7.7), we observed relatively strong and mostly significant negative 
associations with female gender in all countries except Finland, where a positive relationship 
was observed (Model 1). On average we recorded a net effect of about -12 score points. 
When comparing these results with the (mostly smaller) gender differences reported in Chapter 
4, readers should be mindful that the coefficients presented here are those estimated within 
schools and after controlling for other variables in the model. Typically, gender effects remained 
unchanged after taking other ICT-related factors into account (Model 3). However, in some 
countries the negative coefficients were somewhat stronger than in Model 1. 


Use of test language at home had statistically significant positive effects on CT score in four 
countries (Chile, Finland, France, and Germany) and the benchmarking participant North 
Rhine-Westphalia (Germany) in both Model 1 and Model 3. Expected university education was 
consistently a significant positive predictor of CT (similar to the results for CIL), with the exception 
of Germany. On average we observed a net effect of almost 24 score points, which was only 
slightly lower for Model 3 where ICT-related factors were considered as well. 


The within-school effects of socioeconomic background were significant and positive in all 
countries, on average across participating countries one national standard deviation was 
associated with a change of almost 15 score points in Model 1 and with over 12 points in Model 
3. However, they ranged from less than 10 points in Germany to more than 20 score points 
in France. The effects at the school level were significant and positive in all countries except 
Denmark and Korea, in both models. For Model 1, we recorded effects of about 30 score points 
on average, with significant effects ranging from about 21 points in Finland to almost 88 points 
in Germany. 


When reviewing the effects of ICT-related variables on CT at the student level (Table 7.8), we 
noted similar results to those found for CIL. The most consistent positive predictor was students’ 
daily use of ICT, associated with more than 33 points on average, ranging from about 20 (in 
France) to almost 48 score points (in Korea). These coefficients were only slightly smaller for 

odel 3 after controlling for personal and social background factors, with an average effect 
of 31 points. Having five or more years of experience with computers was also a positive and 
significant predictor in all countries except Germany, with an average effect of almost nine score 
points in Model 2 (and about eight points for Model 3). Larger effects for this predictor of 10 
or more score points were observed in Denmark, Finland, and Korea. 


Having two or more computers at home had significant positive effects for Model 2 in all 
countries except Germany and Portugal. On average across participating countries, we observed 
an effect of about 12 score points. However, after controlling for factors related to personal and 
social background, for Model 3 this predictor was only significant in Finland with a considerably 
smaller effect coefficient. In Portugal, for Model 3 we recorded a significant negative effect after 
controlling for personal and social background factors. 


Student report on the use of general ICT applications in class had significant positive associations 
within schools with CT in Denmark, Finland, and France. For Model 2 in these countries one 
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CHAPTER 8: 


Reflections on the IEA International 
Computer and Information Literacy Study 
2018 


In this chapter we reflect on the contribution of the International Computer and Information 
Literacy Study (ICILS) to understanding the extent to which young people are able to use 
information and communication technology (ICT) productively for a range of purposes. Those 
purposes relate to what happens in school and in other environments such as home, society, 
and future workplaces. In addition, we provide perspectives on factors that are associated with 
development of the digital capacities of young people. We provide an overview of the main 
themes emerging from the analyses presented in this report. Within each theme, we reflect 
on the implications of these results for developments in policy and practice that support the 
teaching and learning of ICT-related competencies in schools. 


ICILS as a pioneering study 


ICILS 2013 was the first cycle of the study. It established an agreed definition and explication 
of computer and information literacy (CIL) as a student learning outcome. Furthermore, it 
operationalized a standardized approach to the measurement of CIL in an international large- 
scale assessment (ILSA), and established measures of CIL that could be used to monitor and 
investigate a critical aspect of students’ digital competence. 


ICILS 2018 has extended the work of ICILS 2013 in investigating grade 8 students’ capacities 
to use ICT productively for a range of purposes, including those that go beyond a basic use 
of computers. ICILS 2018 updated the digital contexts for the measurement of students’ CIL 
and investigated in greater detail the nature of classroom teaching activities associated with 
the development of CIL. In addition, it developed and implemented an objective assessment of 
computational thinking (CT). While concepts associated with CT have been recognized since 
the advent of computing as a field of study in the 1940s (Denning 2017), the role of CT in 
curricula has waxed and waned since the introduction of personal computers in schools in the 
final quarter of the twentieth century. In recent years, there has been a resurgence of interest in 
the teaching and learning of CT as both a foundation for the effective use of digital technologies 
and as a transferable set of problem solving skills. |CILS 2018 included an option for countries to 
collect objective data measuring students’ capacities to plan and operationalize computer-based 
solutions to real-world problems. ICILS 2018 established both a framework for measuring and 
reporting achievement in CT and baseline measures against which CT can be monitored over 
time. ICILS remains at the forefront of innovation in the measurement and research of digital 
literacy related competences in ILSA. 


The nature of CIL and CT 
Describing CIL 

ClL refers to an “individual’s ability to use computers to investigate, create, and communicate in 
order to participate effectively at home, at school, in the workplace, and in society” (Fraillon et al. 
2013, p. 17). In ICILS, there is an emphasis on students’ abilities to use computer technologies 
to collect and manage information and to produce and exchange information. 


The CIL achievement scale was established with a midpoint (average) of 500 and with 100 scale 
points as one standard deviation of achievement across ICILS 2013 countries. Theoretically, 
the scale is “unbounded” at the top and bottom allowing for the measurement and description 
of extremely high and low CIL achievement. The scale has been described using four levels of 
achievement. Each level spans 85 scale points and the described levels of the scale range from 
Level 1 (from 407 to 491 scale points) to Level 4 (above 661 scale points). The levels were 
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established such that when a student’s achievement is reported to be within a given level, we 
can be confident that the level describes the CIL that can be demonstrated by the student, even 


if the student’s CIL scale score is near th 
can be assumed to have mastered conten 


e bottom of the level. Students within any given level 
t at lower levels. 


Students working at Level 1 (from 407 to 491 scale points) demonstrate familiarity with the 
basic range of software commands that enable them to access files and complete routine text 


and layout editing under instruct 


ion. They recognize not only some basic conventions used by 


electronic communications software, including knowing which communication tool to use in a 


given context, but also the 


potential for misuse of computers by unauthorized users. 


Students working at Level 2 (from 492 to 576 scale points) demonstrate basic use of computers 


as information resources. | 


select and add content to 


informati 


[hey are able to locate explicit information in simple digital resources, 
on products, and exercise some control over laying out and 


formatting text and images in information products. They can explain the advantage of using a 


given communication tool 


nformati 


on. 


a 


access to some electronic in 


Students working at Level 3 ( 
and understanding to independently search 


ectronic resources, and the information prod 
layout and design. Furthermore, students wor 
information they access may be biased, inaccurate, o 
weaknesses of the use of a given communicatio 


ing at 


information 


in a given context and demonstrate awareness of the need to protect 
formation and of possible consequences of unwanted access to 


from 577 to 661 scale points) possess sufficient knowledge, skills, 
or and locate information. They also have the 
ability to edit and create information products. They can select relevant information from within 
ucts they create exhibit their capacity to control 
Level 3 demonstrate awareness that the 
r unreliable. They also can evaluate the 
n tool ina given context. 


Students working at Level 4 (above 661 scale points) execute control and evaluative judgment 
when searching for information and creating 
awareness of audience and purpose when sea 
include in information products, and formattin 
create. Students working at Level 4 additionally dem 


products. They also demonstrate 


rching for information, selecting information to 
g and laying out the information products they 
onstrate awareness of the potential for 


information to be a commercial and malleable commodity and apply the conventions of a given 
communication tool in a given context to support inclusivity. 


Describing CT 


In ICILS 2018, CT is defined as “an individual’s ability to recognize aspects of real-world 


problems which are approp 


of CT is the first time that an obj 


lection in an ILSA. 


riate for computational formulation and to evaluate and develop 
algorithmic solutions to those problems so that the solutions could be operationalized with a 
computer” (Fraillon et al. 2019, p. 27). In ICILS there is an emphasis in CT on students’ abilities 
to conceptualize problems, plan and evaluate solutions, and to operationalize solutions using 
algorithms and simple computer code. 


The ICILS 2018 assessmen 
created and used in data co 


ective assessment of CT has been 


Data collected in 2018 were used to establish 


the CT achievement scale. The scale was centered around a midpoint of 500 (determined using 
the average achievement of the ICILS 2018 countries partic 
with 100 scale points representing one standard deviation of achievement across the ICILS 
2018 countries that participated in the CT assessment. 


ipating in the CT assessment) and 


The CT scale has been described according to three regions of achievement. The primary 
purpose of these regions is to describe the increase in complexity of CT knowledge, skills, and 
understandings from low to high achievement in order to understand learning progress in this 


newly defined area. 
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Students showing achievement corresponding to the lower region of the scale demonstrate 
familiarity with the basic conventions of digital systems to configure inputs, observe events, 
and record observations when planning computational solutions to given problems. When 
developing solutions in the form of algorithms, they can use a linear (step by step) sequence of 
instructions to meet task objectives. 


Students showing achievement corresponding to the middle region of the scale demonstrate 
understanding of how computation can be used to solve real-world problems. They can plan and 
execute systematic interactions with a system so that they can interpret the output or behavior 
of the system. When developing algorithms, they use repeat statements effectively. 


ica) 


tudents showing achievement corresponding to the upper region of the scale demonstrate an 
nderstanding of computation as a generalizable problem-solving framework. They can explain 
how they have executed a systematic approach when using computation to solve real-world 
problems. Furthermore, students operating within the upper region can develop algorithms that 
use repeat statements together with conditional statements effectively. 


Cc 


CIL and CT achievements vary greatly within countries 


There were large variations that existed in student CIL and CT proficiency within countries. We 
will address variations in CIL and CT in turn. 


Variations in CIL 


In ICILS 2018, the average CIL achievement of students across countries varied from 395 scale 
points to 553 scale points, which spans from Below Level 1 to within Level 3 (Table 3.4). Of 
particular interest in ICILS were the variations in CIL achievement within countries. Even within 
the countries with higher average CIL achievement there were many students who were able 
to demonstrate only the basic functional skills described in CIL Level 1. Students working at 
this level can for example: 


e |dentify who receives an email by carbon copy (CC); 


e Identify problems that can result from mass messaging; 

e Record key points from a video into a text-based note taking application; 
e Use software to crop an image; 

e Place a title in a prominent position on a webpage; 

e Create a suitable title for a slide show; 

e Insert an image into a document; and 


e Suggest one or more risks of failing to log out from a user account when using a publicly 
accessible computer. 


In ICILS 2018 more than one quarter of students in all but one country and one benchmarking 
participant were performing at Level 1 or Below Level 1. In five countries, the proportion was 
more than one half (Table 3.5). So regardless of variations in CIL achievement across countries, 
there remained large numbers of students who were able to complete only the most basic 
functional tasks on a computer. 


With this in mind, across all countries there were students who were achieving at the higher levels 
on the CIL scale. In three countries and one benchmarking participant more than one quarter 
of students were performing at Level 3 or above (Table 3.5). As a minimum these students are 
able to, for example: 


e Identify when content published on the internet may be biased as a result of a publisher's 
content guidelines or advertising revenue directing content; 
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e Select relevant information according to given criteria to include in a website; 


e¢ Know what information is useful to include when recording a source of information from the 


internet; 


e Use generic online mapping software to represent text information as a map route; 


e Select an appropriate website navigation structure for given content; 


e Select and adapt some relevant information from given sources when creating a poster; 


e Demonstrate control of color and contrast to support 


e Demonstrate control of text layout when creating a presentation. 


While variations in CIL existed across countries, the variation within countries was large. Within 
each ICILS 2018 country or benchmarking participant the difference in achievement between 
the top five percent of students and the bottom five percent of students was more than 200 


scale points (Table 3.4). This within country difference is 
achievement between the highest and lowest achieving 


variations in achievement within countries and the proportions of students demonstrating only th 


most basic CIL skills suggest that acquisition of these skil 


InICILS 2013, we questioned the relevance of the description of the current generation of young 


people as digital natives. The ICILS 2018 data again sup 


readability of a poster; and 


larger than the difference in mean CIL 
countries. The existence of these large 
e 


s cannot be left to incidental learning. 


port the contention that, regardless of 


our own impressions of the facility with which young people embrace new technologies, there 
remain large proportions of young people who can complete only the most basic technical 
operations when using a computer. One danger of assuming that young people are imbued with 


a capacity to manage complex functions on computers 


is that we may infer that there is little 


need to formally address knowledge, skills, and understandings as part of schooling. The ICILS 
2018 data support the findings of ICILS 2013 in demonstrating that many grade 8 students have 
developed little more than rudimentary CIL capacities. Formal schooling needs to play a more 
significant role in developing these capacities within young people given that it is clear that, for 
many students, they are not currently being developed through other means. 


Variations in CT 


In ICILS 2018, the average CT of students across countries varied by 76 scale points, from 460 


to 536 scale points, all of which are in the middle region 


of the scale. However, within countries 


there was considerable variation in student CT. In all countries the difference in CT between the 


top and the bottom five percent of students was greater 


than 250 scale points, more than three 


times the difference between the highest and lowest average CT across countries (Table 4.1). 


In all but two countries more than one quarter of students’ CT scores were less than 459 


scale points. This is 
students were, at most, able to: 
e 


¢ Create acomp 


described as CT achievement in the lower region of the CT scale. These 


te but suboptimal route from one location to another on a network diagram; 


e Partially debug an algorithm that uses a repeat statement by correcting the logic of connected 


statements; 


e Create an efficient algorithm that meets all of the given task objectives for a low-complexity 
problem (i.e., a problem with a limited set of available commands and objectives); and 


e Create an inefficient algorithm that meets all of the given task objectives for a medium- 
complexity problem (e.g., a problem with multiple objectives best solved using a repeat 


statement). 
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As was reported for CIL achievement, regardless of variations in achievement across countries, 
there were large proportions of students who are able to complete only the most basic CT- 
related tasks. Where tasks were more complex, such as those with simple iterative logic or with 
multiple solutions, these students would typically provide incomplete or inefficient solutions. 


Across countries, between two and 16 percent of students had CT achievement scores of 
more than 639 scale points, or in the upper region of the scale. These students were likely to 
be able to: 


e Explain the value of a digital system for real-world problem solving; 
e Complete a simple decision tree with the correct use of both logic and syntax; 


e Debug, with the most efficient solution, an algorithm for a high-complexity problem (e.g., a 
problem with multiple task objectives best solved using repeat and conditional statements); 
and 


e Create an efficient algorithm that meets all of the objectives for a high-complexity problem 
e.g., multiple task objectives best solved using repeat and conditional statements). 


It is encouraging to note that some students, albeit in relatively small proportions, demonstrated 
understandings of the use of digital systems in real-world problem solving and were able to 
create efficient algorithms expressed as visual code commands. The broad range of student CT 
achievement within countries shows that these skills can be developed by grade 8 students but 
suggests that these skills may not be developed through exposure to and use of digital devices 
alone. In Chapter 2, we reported considerable variation across countries regarding their national 
curriculum emphases on CT although most countries do provide a separate ClL-related subject 
that includes at least some coding. There is opportunity across countries to increase the emphasis 
on the core aspects of CT in curriculum and learning programs. 


CIL, CT, digital literacy, and student gender 


In ICILS 2018 CIL and CT are conceptualized, operationalized, measured, and reported as 
separate areas of learning. The two domains are clearly discrete in their focus. CIL emphasizes 
information literacy in receptive and productive communicative contexts, whereas CT emphasizes 
problem specification and the creation of solutions that can be implemented by computers. 
Despite the obvious conceptual and operational differences, it is clear that the two domains 
are practically linked because both involve interaction with digital devices. At a conceptual level, 
the domains are both rooted in an understanding of how computers process the information 
we provide them, how they are used as tools, and the conventions associated with computer 
use and software environments. 


These two ICILS domains can be regarded as complementary aspects of a broader notion of 
digital competence. However, the complementarity of the two domains and their contribution to 
a larger notion of digital competence remains a question for further investigation. For example, 
the CIL content in ICILS can be seen to relate to each of the five competence areas (information 
and data literacy; communication and collaboration; digital content creation; safety; and problem 
solving) described in the European Commission’s DigComp 2.0 Framework (Vuorikari et al. 
2016). The ICILS CT content domain can be regarded as particularly relevant to the DigComp 
problem-solving competence area. While it is possible to see how the two ICILS achievement 
domains can be considered under an example of a broader notion of digital competence, the 
Computational Thinking Study under the EU Science Hub, lists the relationship between CT and 
digital literacy/digital competence as one of the areas in which more evidence-based research 
is needed (European Commission 2019). ICILS 2018 data and data to be obtained from future 
cycles of ICILS may contribute to better understanding of this connection. Under an alternative 
approach, the US National Assessment of Educational Progress Technology and Engineering 
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Literacy assessment includes content that overlaps with each of CIL and CT although without the 


same emphasis on CT as a form of problem solving processes (see, NAGB [National Assessment 


Government Board] 2014). 


ICILS 2018 has provided empirical data that contributes to our understanding of the relationship 
between CIL and CT. Achievement in the two domains were highly correlated. Across ICILS 


countries that participated in CIL and CT the correlation in achievement ra 


0.89. The high correlation between CIL and CT may be explained, in part, by the relationship 


of achievement in each domain to general academic ability. However, d 
association between CIL and CT achievement, data from ICILS 2018 show di 
achievement in each of CIL and CT by student gender. Female students sh 
higher CIL achievement than male students, and male students tended to d 
CT achievement than female students. 


If one accepts the proposition that CIL achievement relates to information li 


ICT context and CT achievement relates more to specialized ICT use, then the difference in the 
association between each of CIL and CT and student gender is consistent with the patterns of 
gender difference in students’ uses of and attitudes towards the use of ICT. Female student 
tended to report using ICT for school-related purposes and using general ICT applications slight 


more than male students. Where differences occur, female students tended 


more about ClL-related content at school than did male students and overall there was litt 
reported difference in students’ general ICT self-efficacy. However, male students reported mor 
than female students that they were learning about CT-related content at school. Male students 
also reported consistently higher ICT self-efficacy regarding specialist ICT tasks, generally more 
positive perceptions about the role of ICT in society, and generally less negative perceptions 


about the role of ICT in society. 


bout the differences in female and male students’ attitudes towards and u 


from within digital sources and creating digital artefacts to communicate info 
Male students are more confident to approach, and slightly stronger at deal 


nged from 0.74 to 


espite the strong 
fferent patterns of 
owed consistently 
emonstrate higher 


teracy in a general 


S 
y 
to report learning 
e 
e 


ses of ICT. Female 


The ICILS 2018 data regarding CIL, CT, and student gender are consistent with current beliefs 
a 
S 


tudents are stronger users of ICT for general school-related tasks (such as locating information 


rmation to others). 
ing with, specialist 


ICT tasks (such as changing computer settings or creating programs). These findings provide 
a challenge to curricular and educational policy. If CIL and CT are valued by policymakers as 
components of a broader digital literacy necessary for effective participation in an increasingly 


digital world, then ICILS 2018 raises questions about how these gender diffe 
and achievement may be addressed. 


Evidence of the digital divide 


The notion of a digital divide is, at its simplest, a reference to the varying 


infrastructure; the use of technology in the classroom; and the use of techn 
individuals. While ICILS 2018 was not designed with a particular model of t 


digital divide can extend beyond access to technology to include how tec 
schools and how students are empowered through technology to participate in 


Twelve of the 14 1CILS 2018 countries and benchmarking participants indicate 


rences in attitudes 


opportunities and 


access that people have to ICT. Hohlfeld et al. (2008) postulated a model for researching the 
digital divide in schools, which can be considered to have three levels: access to technological 


ology to empower 
he digital divide in 


mind, for the purpose of this discussion we accept the proposition that consideration of the 


hnology is used in 
their digital world. 


d that reducing the 


digital divide between groups of students was emphasized in their plans, policies, and priorities 
regarding the use of ICT (see Chapter 2). In ICILS 2018, the relationships between student 
socioeconomic status (SES) and CIL and CT were consistent and clear. Across ICILS countries 
student SES (both for individual students and averaged within their schools) was a strong 


REFLECTIONS ON IEA ICILS 2018 245 


predictor of student CIL and CT. Other consistent predictors were students’ experience of using 
computers and their frequency of using ICT at home (Tables 3.8, 3.10, 4.3, 4.5, 7.5, and 7.10). 


While the relationship between student SES and educational achievement is a consistent and 
pervasive finding in ILSA studies across all learning areas, in ICILS the relationship of SES to CIL 
and CT appears to involve a combination of factors. The generally observed outcome of SES- 
related educational advantage or disadvantage combines with the specific SES-related influences 
relating to the digital divide. Students from low SES backgrounds may have limited access to ICT 
infrastructure and the educational benefits associated with its use in learning about CIL and CT. 


It is beyond the scope of this report to investigate in detail the nature and effect of the digital 
divide within and across individual ICILS countries. However, evidence of a digital divide is 
clearly apparent in the student achievement results in both CIL and CT. This finding alone 
warrants consideration about how countries may work to reduce the divide. The ICILS data 
ffer the potential to examine evidence of a digital divide within countries, not just in terms of 
infrastructure provision and access to ICT but to probe differences in approaches to the teaching 
nd learning of CIL and CT in schools. The ICILS teacher and school-level data offer a resource 
to be investigated from this perspective. 


[e) 


ied) 


Digital devices as digital textbooks 


CILS 2018 called into question “the idealized images commonly associated with visions of 
CT in teaching and learning” (Fraillon et al. 2014, p. 257). One of these images related to the 
degree to which digital technologies were being used to transform classrooms from traditional 
teacher-centered environments into more constructivist learning environments. In ICILS 2013, 
we found that the most commonly reported uses of computers in classes by students related 
to tasks associated with document preparation and presentations. The activities reported most 
frequently by teachers related to presenting information to students and skills development 
through repetition. The conclusion in 2013 was that “computers were most commonly being 
used to access digital textbooks and workbooks rather than to provide dynamic, interactive 
pedagogical tools” (Fraillon et al. 2014, p. 257). 


nICILS 2018 we again sought to investigate the ways in which students and teachers perceived 
the use of ICT in teaching and learning. As the countries that participated in ICILS 2018 are 
argely different to those that participated in ICILS 2013, we will not discuss changes within 
countries, rather we will discuss the findings of ICILS 2018 (reported in detail in Chapters 5 and 
6) as a snapshot of ICT use in classes across a broad range of countries in 2018. The context for 
this can be compared to that of ICILS 2013 in which a similar snapshot was made, albeit across 
asomewhat different but similarly broad set of countries. 


In ICILS 2018 the activities using ICT applications reported most frequently by students in 
their lessons were using computer-based information resources, word-processing software, and 
presentation software. The least frequently reported activities were using simulations and modeling 
software, concept mapping software, and tools that capture real-world data (Table 5.17). 


The general ICT utility tools reported most frequently used by teachers in most lessons were 
word-processor software, presentation software, computer-based information resources, and digita 
contents linked with textbooks. Each of these was reported, on average across countries, to be used 
in most lessons by more than 30 percent of teachers. In comparison, teachers reported less use 
of digital learning tools in class (Table 6.20). Learning management systems and interactive digita 
learning resources were the digital learning ICT tools reported on average across all countries 
by more than 20 percent of teachers to be used in most lessons. The least frequently reported 
digital learning tools were practice programs or apps, e-portfolios, concept mapping software, and 
simulations and modeling software, all of which were reported to be used in most lessons by fewer 
than 10 percent of teachers (Table 6.21). 
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High proportions of students did, however, report having learned to a large or moderate extent a 
range of content and processes associated with each of CIL and CT. On average across countries, 
ClL-related activities were reported to have been learned about at school, to a large or moderate 
extent, by more than 60 percent of students (Table 5.21). Only two aspects of CT (to make flow 
diagrams to show the different parts of a process and to use simulations to help understand or solve 
real-world problems) were reported to have been learned at sc 


by fewer than 50 percent of studen 


tS ON average across COUN 


Students’ reported experience of learning about CIL and CT i 
reports of their emphasis on Cll and CT-related capabilities 


countries more than 60 percent o 


hool, to a moderate or large extent, 
tries (Table 5.24). 


n class was consistent with teacher 
in their classes. On average, across 


f teachers reported placi 


ng some or a strong emphasis on 


ClL-related capabilities with the exception of to provide digital feedback on the work of others, 


which was emphasized by 49 perce 


were less frequently reported to be emphasized by teachers 
than 60 percent of teachers reported placing some or strong emphasis on six of nine CT-related 


capabilities with the remaining three being en 


(see Chapter 6). 


So, while it is positive news that teachers an 


nt of teachers (see Chapter 6). While CT-related capabilities 


than ClL-related capabilities, more 


nphasized by fewer than 50 percent of teachers 


d students agree that CIL and CT learning are 


being included in classes, the picture of classroom use of ICT in ICILS 2018 is similar to that 
reported in ICILS 2013 (albeit in a largely different set of countries). Despite the passage of time 


between 2013 and 2018 and the concomitan 


resources for teaching and learning, the observation from IC 


education were most commonly being used 
this raises a number of questions about the con 
provided around the use of ICT in teaching and learni 
The questions should not only relate to the reasons for, o 


mismatch between pedagogical rhetoric and classroom real 
extend to what it is reasonable to expect from classroom teachers in their use of ICT in teaching 
and what best practice use of ICT looks like within countries, 


Schools and teachers should not be offered u 
ICT use in classrooms. However, further work needs 


presented to schools and teachers 
of ICT in teaching. 


t increase in the availability of digital technology 


ILS 2013 that computers in school 


as digital textbooks remains valid. For policymakers 
trast between the messages that are frequently 
ng and the reality of classroom practice. 


r even the detailed nature of, the 
ities. Rather, the questions should 


schools, and across learning areas. 


nrealistic and unachievable idealized images of 
to be done to determine what should be 


as desirable, productive, and even “best practice” in the use 


Supporting teachers to use ICT in their teaching 


We can see from the data reported in Chapter 2 that the development of Cll and CT-related 
competencies are valued across ICILS countries. They are emphasized within educational plans 


and policies and manifest in countries’ commitme 


learning support for teachers, and 


nts to providing infrastructure, professional 


learning materials relating to these areas. While CIL has a 


greater emphasis in curriculum across countries than CT, bo 
of most ICILS countries. This commitment across countries to developing digital literacy related 


competencies brings with it the question of what 
about ICT (see Chapter 6) can further tell us about 
ICT in their teaching and to facilitate student learni 


Provision of ICT 


th are represented in the curricula 


the ICILS 2018 data on teaching with and 
how teachers can best be supported to use 
ng in ClL and CT. 


Across the ICILS 2018 countries, teachers were generally well experienced with the use of ICT 
in their preparation and delivery of lessons and overall indicated that they were confident to use 
ICT in their teaching for a broad range of actions. However, despite these positive indicators, 
on average across all ICILS countries, fewer than half the teachers reported using ICT in their 
teaching every day, and in only one country was ICT use reported to be used in teaching daily 
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by more than 70 percent of teachers (Table 6.1). This raises the additional question of what can 
be done to increase the use of ICT in teaching. 


Unsurprisingly, we found that teachers were more likely to use ICT in teaching if they believed 
they were in a school where there were sufficient ICT resources available to them. In this case, 
ICT resources referred to infrastructure, technical support, time to plan lessons, and having 
the opportunity to develop expertise in ICT (Table 6.10 and 6.13). It is interesting to note that 
teachers perceived the practical support associated with the provision of time and opportunity 
to develop skills as congruent with the provision of ICT infrastructure. This suggests that, while 
the provision of ICT infrastructure in schools can impact on the likelihood of teachers using 
ICT, they should be accompanied with the provision of time for teachers to plan for ICT use 
and develop ICT skills. When developing ICT plans and policies, it may be valuable to consider 
teacher time and opportunities to learn as aspects of infrastructure rather than as separate 
aspects of policy and planning. 


Collaborative school environment 


nICILS 2013 teachers’ perceptions of the degree to which they were in a collaborative school 
environment regarding the implementation of |CT emerged as a factor relating to teachers’ 
ikelihood to emphasize CIL in their teaching. In ICILS 2018, across a largely different set of 
countries, teachers’ perceptions of collaboration regarding the use of ICT was again an important 
factor in explaining teacher practice. 


n ICILS 2018, while the student assessment of CT was optional for countries, the teacher 
questionnaire completed by all countries included a question relating to teachers’ emphasis on 
CT-related tasks in their teaching. In all ICILS 2018 countries, teacher perceptions of collaboration 
regarding the use of ICT was associated with their emphasis on developing both Cll and CT- 
related outcomes in their students (Tables 6.17 and 6.19). 


This finding was consistent with that of ICILS 2013. Across the two cycles of ICILS we can see 
that teacher responses to their perception of working in a collaborative school environment have 
been expressed across a broad range of countries and with reference to their teaching of CIL and 
CT. To this effect we can see that the findings of ICILS 2018 have confirmed and strengthened 
those of ICILS 2013 and should be given due attention by policymakers when planning how 
to support the work of teachers in schools. Teachers are providing the clear message that their 
implementation of CIL- and CT-related content in their teaching is advantaged by the sense that 
they are working in a school with a collaborative approach to the use of ICT. 


Nurturing confident and enthusiastic teachers 


In ICILS 2018 we found that teachers’ beliefs in their own capacities to use ICT and also their 
beliefs about ICT are positively associated with each other and to the degree of emphasis they 
place on CIL and CT in their teaching. Teachers with higher ICT self-efficacy were more likely 
to hold more positive views about the use of ICT in teaching and less negative views about the 
limitations of ICT use (Table 6.7). From this we can develop an image of teachers who are confident 
and vigorous proponents of ICT in education in comparison to those who lack confidence and 
belief in the pedagogical value of ICT. Ideally, education systems would work to support the 
development of the former profile in their teachers. 


While teacher confidence to complete tasks using ICT was generally high, this confidence did 
vary across countries but more importantly with age within countries. In all countries, teachers 
under the age of 40 expressed higher levels of ICT self-efficacy than teachers aged 40 years 
or older (Table 6.3). The ICILS 2018 data raise the question of whether targeted programs to 
support the development of ICT use by older teachers may help to redress this imbalance. 
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APPENDIX A: 


Sampling information and participation rates 


Table A.1: Coverage of ICILS 2018 target population 


Country International target Exclusions from target population (%) 
population coverage (%) At school level Within sample Overall 
Chile 00 0.5 0.8 1.3 
Denmark 00 3.1 44 12) 
Finland 00 1.6 2.4 4.0 
France 00 3.4 1.3 47 
Germany 00 15 2.9 43 
taly 00 0.1 2.9 3.0 
azakhstan 00 3.4 21, 5.6 
orea 00 0.9 0.6 1.5 
Luxembourg 00 0.0 3.9 3.9 
Portugal 00 0.8 8.0 8.9 
United States 00 0.0 5.0 5.0 
Uruguay 00 1.41 0.0 iil 
Benchmarking participants 
oscow (Russian Federation) 00 0.7 23 3.0 
orth Rhine-Westphalia (Germany) 00 1.4 31 4.6 


Note: Results are rounded to one decimal place. 


© IEA International Association for the Evaluation of Educational Achievement 2020 
J. Fraillon et al., Preparing for Life in a Digital World, 
https://doi.org/10.1007/978-3-030-3878 1-5 
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APPENDIX C: 


Percentiles, means, and standard deviations 
of computer and information literacy and 
computational thinking 


Table C.1: Percentiles of computer and information literacy 
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Country 5th percentile 25th percentile 75th percentile 95th percentile 
Chile 325 (11.3) 425 (6.2) 534 ( 600 (5.7) 
Denmark? * 435 6.0) 515 (3.0) 598 ( 650 (3.9) 
Finland 381 (7.6) 486 (4.2) 587 ( 645 (4.0) 
France 357 9.6) 448 (4.0) 555: °( 618 (38) 
Germany 372 9.0) 470 (4.7) 574 ( 634 (4.3) 
azakhstan! 217 (9.8) 322 ~=—(7.7) 471 (68 564 (8.9) 
orea, Republic of 370 7.1) 481 (4.9) 610 ( 682 (48) 
Luxembourg 329 3.9) 428 (1.9) 542 ( 606 (2.1) 
Portugaltt * 392 = (8.3) 472 (4.0) 565 ( 623 (3.3) 
Uruguay 280 6.8) 380 = (5.5) 527 ( 605 (7.0) 
Testing at the beginning of the school year 
taly 315 (6.8) 410 (5.6) 518 (2.7 587 (3.8) 
Not meeting sample participation requirements 
United States 373 (4.9) 469 (3.1) 576 (2.2 638 (3.1) 
Benchmarking participants meeting sample participation requirements 
oscow (Russian Federation) 425 (7.3) 509 = (2.8) 595, (25 651 = (3.7) 
orth Rhine-Westphalia (Germany) 385 (7.5) 467 (3.9) 565 (3.2) 629 (3.8) 


Notes: Standard errors appear in parentheses. 
+ Met guidelines for sampling participation ra 


es only after replacement schools were included. 
it Nearly met guidelines for sampling participation rates after replacement schools were included. 


1 National defined population covers 90% to 95% of the national target population. 
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Table C.3: Percentiles of computational thinking 


Country 5th percentile 25th percentile 75th percentile 95th percentile 
Denmarkt * 385 (7.3) 475 (4.1) 585 (2.8) 661 (4.2) 
Finland 331 (10.0) 449 (4.7) 575 (5.0) 652 (5.3) 
France 342 = (8.3) 441 (3.2) 566 (2.4) 646 8) 
Germany 305 (10.1) 421 (6.3) 553 (4.7) 649 (7.5) 
orea, Republic of 332 (13.3) 471 (5.8) 612 (4.2) 702 (6.7) 
Luxembourg 273 = (3.5) 391 (2.3) 533 (1.5) 628 (3.6) 
Portugal!t * 341 (8.3) 431 (3.1) 535 (3.5) 607 (4.9) 
Not meeting sample participation requirements 
United States 313° (5:1) 429 (3.6) 573 (2.9) 668 (5.2) 
Benchmarking participant meeting sample participation requirements 
orth Rhine-Westphalia (Germany) 325 (6.7) 425 (44) 547 (3.9) 638 (7.2) 
Notes: Standard errors appear in parentheses. 
1 Met guidelines for sampling participation rates only after replacement schools were included. 


iT Nearly met guidelines for sampling participation rates after replacement schools were included. 
1 National defined population covers 90% to 95% of the national target population. 
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APPENDIX D: 


Pair-wise comparisons of average 


achievement data 


Table D.1: Pair-wise comparisons of average computer and information literacy scores 


Country Average 
scale score 
re) 
& 
a 
4 2 “ 2 % 
& | § =| = 2 | 3 
s/ S| pe|s| S|] e/2 S| se 
oO) ee ee |e 2) els 
| 2 te | Gh | |e | SS || S 
Denmark’ * 553 (2.0) AI/A/S/A/,L| AIL AILA/LA/ILAILA 
orea, Republic of Lie ee Vv Al\|A/ | A/I/|A/SL/A/]A/SAIA 
Finland 531 (3.0) viv A\|A|A/S/AIA/A/SA 
Germany Silein (229) viviv A/A/JA/AIA 
Portugaltt * 516 (2.6) viviv A\|A/A/ AIA 
France AOD (2:3) VF | XZ I] 4 M4 | AZ A;|A/,/A/JA 
Luxembourg 482 (0.8) vVvivivivivi Vv Al A 
Chile 476 (3.7) Vf | AZ | AZ I AA AA | M4 A|A 
Uruguay 450 (4.3) viviviviviv A 
azakhstan* 395 (5.4) viviviviviv Vv 


A Achievement significantly higher than in comparison country 
Vv Achievement significantly lower than in comparison country 


Notes: Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals 


may appear inconsistent. 


+ Met guidelines for sampling participation rates only after replacement schools were included. 
it Nearly met guidelines for sampling participation rates after replacement schools were included. 
1 National defined population covers 90% to 95% of national target population. 
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Table D.2: Pair-wise comparisons of average computational thinking scores 


Korea, Republic of 536 (44 A Al|A/A 
Denmark' * bay (oe A/A/A/AJA 
Finland 508 (3.4 viv A;/A/A 
France 501 (24 \é AZ A| AIA 
Germany 486 (3.6 viviviv A 
Portugaltt * 482 (2.5 viviviv A 
Luxembourg 460 (0.9 viv Viv 


A Achievement significantly higher than in comparison country 
Vv Achievement significantly lower than in comparison country 


Notes: Standard errors appear in parentheses. Because results are rounded to the nearest 

whole number, some totals may appear inconsistent. 

+ Met guidelines for sampling participation rates only after replacement schools were 
included. 

‘t Nearly met guidelines for sampling participation rates after replacement schools were 
included. 

+ National defined population covers 90% to 95% of national target population. 
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Student percentages for dichotomous variables 
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APPENDIX F: 
Item maps 


ICILS 2018 used sets of student and teacher questionnaire items to measure constructs relevant 
to the learning context for students’ acquisition of CIL and CT, and use of ICT for teaching and 
learning. Typically, this information was obtained using sets of Likert-type items with more than 
four categories (for example, “strongly agree,’ “agree, “disagree,” and “strongly disagree’) or other 
types of rating scales (for example, “never,” “less than once a month,’ “at least once a month 
but not every week,’ “at least once a week but not every school day,’ and “every school day”). 
The responses to the items were then recoded so that the scale scores reflected the strength 
or frequency of the attitudes or perceptions that were measured. 


We used the Rasch partial credit model (Masters and Wright 1997) for scaling and the resulting 
weighted likelihood estimates (Warm 1989) were transformed into a metric with a mean of 50 
and a standard deviation of 10 for equally weighted ICILS 2018 national samples that satisfied 
guidelines for sample participation. Further details about the scaling and equating procedures 
will be provided in the ICILS 2018 technical report (Fraillon et al. 2020). 


The resulting ICILS 2018 scale scores can be interpreted with regard to the average across 
countries participating in this study, but they do not reveal the extent to which students endorsed 
the items used for measurement. However, our application of the Rasch partial credit model 
lows us to map scale scores to item responses. Thus, it is possible for each scale score to 
predict the most likely item response for a respondent. (For an application of these properties 
in the previous survey, see Schulz and Friedman 2015.) 


ied) 


This appendix provides item maps for each questionnaire scale presented in the report. The maps 
provide a prediction of the minimum coded score (e.g., O = “strongly disagree”, 1 = “disagree”, 2 = 
“agree? and 3 = “strongly agree’) a respondent would obtain on a Likert-type item based on their 
questionnaire scale score. For example, it can be predicated that students with a certain scale 
score have a 50 percent probability of at least agreeing (or strongly agreeing) with a particular 
item (see example item in Figure F.1). For each item, it is possible to determine Thurstonian 
thresholds; these are the points at which a minimum item score becomes more likely than any 
lower score, and they determine the boundaries between item categories on the item map. 


This information can also be summarized at the scale level by calculating the average thresholds 
across all of the corresponding scaled items. For example, when using four-point Likert-type 
scales, this was typically done for the second threshold, making it possible to predict how likely it 
would be for arespondent with a certain scale score to have (on average across items) responses 
in the two lower or two upper categories. Using this approach for items measuring agreement 
made it possible to distinguish between scale scores for the respondents who were most likely 
to agree or disagree with the average item used for deriving the scale. 


nsome of the reporting tables with national average scale scores, means are depicted as boxes 
that indicate their mean values, plus or minus sampling error, as two color graphical displays 
see, for example, Table 5.32). If national average scores are represented by the darker shaded 
area, on average across items students would have had responses in the respective lower item 
categories (for example, “strongly disagree’ or “disagree”). If these scores are represented by the 
ighter shaded areas, then students’ average item responses would have been in the upper item 
response categories (for example, “strongly agree,’ or “agree’). 
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Figure F.1: Example of questionnaire item map 


Scale scores (mean = 50, standard deviation = 10) 


Item 20 30 40 50 60 70 80 
tem iii 
_ iii 
7 
Item 3 
Strongly disagree Disagree i Agree If Strongly agree 


Example of how to interpret the item-by-score map 


1 Arespondent with score 30 has more than 50% probability to strongly disagree with all 
three items 
2 Arespondent with score 40 has more than 50% probability not to strongly disagree with 


items 1 and 2 but to strongly disagree with item 3 


3 Arespondent with score 50 has more than 50% probability to agree with items 1 and to 
disagree with items 2 and 3 


4 Arespondent with score 60 has more than 50% probability to strongly agree with items 1 
and to at least agree with items 2 and 3 


=) Arespondent with score 70 has more than 50% probability to strongly agree with items 1, 


APPENDICES 


Figure F.2: Item map for the scale reflecting students’ use of general applications for activities 


How often do you use ICT for each of the following 
activities? 


Write or edit documents 


Use aspreadsheet to do calculations, store data, or 
plot graphs (e.g., using [Microsoft Excel ®]) 


Create a simple “slideshow” presentation (e.g., using 
[Microsoft PowerPoint ®]) 


Write or edit documents 


Use a spreadsheet to do calculations, store data, or 
plot graphs (e.g., using [Microsoft Excel ®]) 


Create a simple “slideshow” presentation (e.g., using 
[Microsoft PowerPoint ®]) 


271 
Scores 
20 30 40 50 60 70 80 
Never 
Less than once a month 


fj At least once a month but not every week 


Bh At least once a week but not every day 


IB Every day 
14 26 
29 2D 
14 S55) 


25 


16 


Sum 


100 


100 


100 
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Figure F.3: Item map for the scale reflecting students’ use of specialist applications for activities 


Scores 
How often do you use ICT for each of the following 


activities? 20 30 40 50 60 70 80 


Record or edit videos 


Write computer programs, scripts, or apps 
(e.g., using [Logo, LUA, or Scratch]) 


Use drawing, painting, or graphics software or [apps] im 


Produce or edit music 


Build or edit a webpage 


Never 


Less than once a month 


Hj At least once a month but not every week 
Bh At least once a week but not every day 
IB Every day 


on 
ic 
3 


(o} 


Record or edit videos 28 26 19 9 0 


Write computer programs, scripts, or apps 


3 0 
eg using Logo LUA, or Sat - = 


Use drawing, painting, or graphics software or [apps] 39 27 13 7 00 


(o} 


[Sl 


Produce or edit music 55 15) 10 10 0 


ist 
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Figure F.4: Item map for the scale reflecting students’ use of ICT for social communication 
Scores 
How often do you use ICT to do each of the following 
communication activities? 20 30 40 50 60 70 80 


Share news about current events on social media 


Communicate with friends, family, or other people 
using instant messaging, voice, or video chat (e.g., 
[Skype, WhatsApp, Viber]) 


Send texts or instant messages to friends, family, or 
other people 


Write posts and updates about what happens in 
your life on social media 


Post images or video in social networks or online 
communities (e.g., [Facebook, Instagram, or 
YouTube]) 


Watch videos or images that other people have 
posted online 


Send or forward information about events or 
activities to other people 


Never 


Less than once a month 


Hf At least once a month but not every week 
BB At least once a week but not every day 


I Every day 

Sum 
Share news about current events on social media 27 Wy 20 eal 00 
Communicate with friends, family, or other people 
using instant messaging, voice, or video chat (e.g., 4 5) 16 70 00 
[Skype, WhatsApp, Viber]) 
Send texts or instant messages to friends, family, or 5 5 15 70 00 
other people 
Write posts and updates about what happens in your 34 19 18 14 00 
life on social media 
Post images or video in social networks or online 19 30 53 18 00 
communities (e.g., [Facebook, Instagram, or YouTube}) 
Watch videos or images that other people have 4 4 15 71 00 
posted online 
Sender forward information about events or 59 ie 55 cE 00 
activities to other people 
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Figure F.5: Item map for the scale reflecting students’ use of ICT for exchanging information 


How often do you use ICT to do each of the following 
communication activities? 


Ask questions on forums or [Q&A] websites 


Answer other peoples’ questions on forums or 
[Q&A] websites 


Write posts for your own blog (e.g., [WordPress, 
Tumblr, Blogger]) 


Ask questions on forums or [Q&A] websites 


Answer other peoples’ questions on forums or 
[Q&A] websites 


Write posts for your own blog (e.g., [WordPress, 
Tumblr, Blogger]) 


20 


30 


40 


Scores 


50 60 70 


Never 


Less than once a month 


Hj At least once a month but not every week 
Hi At least once a week but not every day 


IB Every day 
60 ge 8 = 
63 16 ch 


100 


100 


100 


APPENDICES 2/5 


Figure F.6: Item map for the scale reflecting students’ use of ICT for accessing content from the 


internet 
Scores 


How often do you use ICT to do each of the following 
leisure activities? 20 30 40 50 60 70 80 


Search the internet to find information about places 
to goor activities to do 


Read reviews on the internet of things you might 
want to buy 


Search for online information about things you are 
interested in 


Use websites, forums, or online videos to find out 
how to do something 


Never 


Less than once a month 


Hl At least once a month but not every week 
BB At least once a week but not every day 
IB Every day 


Su 


3 


Search the internet to find information about places 


O 
o goor activities to do 


fo) 


14 25) 


Read reviews on the internet of things you might 17 20 25 14 0 


want to buy . 
Read news stories on the internet 15 U7! 00 
Search for online information about things you are 

interested in ° ? SH ste a 
Use websites, forums, or online videos to find out 13 Ae ay 6 00 


how to do something 
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Figure F.7: Item map for the scale reflecting students’ use of ICT for study purposes 


How often do you use ICT for the following school- 
related purposes? 


Prepare reports or essays 


Prepare presentations 


Work online with other students 


Complete [worksheets] or exercises 


Organize your time and work 


Take tests 


Use software or applications to learn skills or a 
subject (e.g., mathematics tutoring software, language 
earning software) 


Use the internet to do research 


Use coding software to complete assignments 
e.g., [Scratch]) 


ake video or audio productions 


Prepare reports or essays 


Prepare presentations 


Work online with other students 


Complete [worksheets] or exercises 


Organize your time and work 


Take tests 


Use software or applications to learn skills or a 
subject (e.g., mathematics tutoring software, language 
earning software) 


Use the internet to do research 


Use coding software to complete assignments 
(e.g., [Scratch]) 


ake video or audio productions 


Scores 


20 30 40 50 60 70 


ioe) 
oO 


Never 


Less than once a month 


Hf At least once a month but not every week 
BB At least once a week but not every day 


IB Every day 

Sum 

18 28 21 6 00 
13 32 19 4 00 
35 22 16 ) 00 
29 pe v0) 4 00 
38 18 16 13 00 
36 24 16 5 00 
34 22 Wi 7 00 
7 13 Sil 8 00 
52 20 aT@) 4 00 
50 Dal aT@) 8 00 
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Figure F.8: Item map for the scale reflecting students’ use of general applications in class 


Scores 
When studying throughout this school year, how often 20 30 40 50 60 70 80 
did you use the following tools during class? 
Word-processing software (e.g., [Microsoft Word ®]) mz 
Presentation software (e.g., [Microsoft PowerPoint ®]) | xm” 
Computer-based information resources (e.g., websites, .—l sm 
wikis, encyclopaedia) 
Never In some lessons 
BB In most lessons IB !nevery or almost every lesson 


Sum 
Word-processing software (e.g., [Microsoft Word ®)]) 23 48 9 100 
Presentation software (e.g., [Microsoft PowerPoint 0 53 5 100 
®)) 
Computer-based information resources (e.g., = 
webstes wikis encyelonaedt) zi E 0 
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Figure F.9: Item map for the scale reflecting students’ use of specialist applications in class 


Scores 


When studying throughout this school year, how often 20 30 AO 50 60 70 80 
did you use the following tools during class? 


Multimedia production tools (e.g., media capture 
and editing, web production) 


Concept mapping software (e.g., [Inspiration ®], 
[Webspiration ®]) 


Tools that capture real-world data (e.g., speed, 
temperature) digitally for analysis 


Interactive digital learning resources (e.g., learning 
games or applications) 


Graphing or drawing software 


Simulations and modeling software 


Never In some lessons 


BB In most lessons IB in every or almost every lesson 


Sum 
Multimedia production tools (e.g, media capture and = a z 00 
editing, web production) 2 
Concept mapping software (e.g., [Inspiration ®], 67 5 vv Ei 00 
[Webspiration ®]) a 
Tools that capture real-world data (e.g., speed, 62 09 00 
temperature) digitally for analysis 
Simulations and modeling software 67 25 ->o im 00 
Interactive digital learning resources (e.g., learning 48 37 00 
games or applications) 
Graphing or drawing software 54 32 00 
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Figure F.10: Item map for the scale reflecting students’ learning of ICT tasks at school 

Scores 
At school, to what extent have you learned how to do the 20 30 40 50 60 70 80 


following tasks? 


Provide references to internet sources 


Search for information using ICT 


Present information for a given audience or purpose 
using ICT 


Work out whether to trust information from the 
internet 


Decide what information obtained from the internet 
is relevant to include in school work 


Organize information obtained from internet 
sources 


Decide where to look for information on the 
internet about an unfamiliar topic 


Use ICT to collaborate with others 


Not at all o.asmall extent 
HB Toa moderate extent BB loalarge extent 
Sum 
Provide references to internet sources 10 oo 26 00 
Search for information using ICT 9 Aw 36 00 
Present information for a given audience or purpose 5 4 % 00 
using ICT 
Work out whether to trust information from the 3 22 00 
internet 
Decide what information obtained from the internet 9 21 29 00 
is relevant to include in school work a 
Organize information obtained from internet sources 7 23 27 00 
Decide where to look for information on the internet 3 04 99 00 


about an unfamiliar topic 


) 
a} 

jo} 

oO 


Use ICT to collaborate with others 7 23 


t 
N 


280 PREPARING FOR LIFE INA DIGITAL WORLD 


Figure F.11: Item map for the scale reflecting students’ learning of ICT coding tasks at school 


Scores 
When studying during the current school year, to what 20 30 40 50 60 70 80 
extent have you been taught how to do the following 
tasks? 


To display information in different ways 


To break a complex process into smaller parts 


To understand diagrams that describe or show real- 
world problems 


To plan tasks by setting out the steps needed to 
complete them 


To use tools to make diagrams that help solve 
problems 


To use simulations to help understand or solve real- 
world problems 


To make flow diagrams to show the different parts 
of a process 


To record and evaluate data to understand and solve 
a problem 


To use real-world data to review and revise solutions 


to problems 
Not at all oasmall extent 
HB to amoderate extent BB loalarge extent 
Sum 
To display information in different ways 8 16 ew 00 
To break a complex process into smaller parts 18 25 00 
To understand diagrams that describe or show real- 49 23 9 00 
world problems 
To plan tasks by setting out the steps needed to 13 3 00 
complete them 
To use tools to make diagrams that help solve 78 28 00 
problems 
To use simulations to help understand or solve real- 17 24 22 00 
world problems 
To make flow diagrams to show the different parts of 28 28 15 00 
a process 
To record and evaluate data to understand and solve 16 2% 20 00 
a problem 
To use real-world data to review and revise solutions 17 24 a) 00 


to problems 
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NO 
(oe) 


Figure F.12: Item map for the scale reflecting students’ self-efficacy regarding the use of general 


applications 


How well can you do each of these tasks when using ICT? 


Edit digital photographs or other graphic images 


Write or edit text for a school assignment 


Search for and find relevant information for a school 
project on the internet 


Create a multimedia presentation (with sound, 
pictures, or video) 


Upload text, images, or video to an online profile 


nsert an image into a document or message 


nstall a program or [app] 


Judge whether you can trust information you find 
on the internet 


Edit digital photographs or other graphic images 


Write or edit text for a school assignment 


Search for and find relevant information for a school 
project on the internet 


Create a multimedia presentation (with sound, 
pictures, or video) 


Upload text, images, or video to an online profile 


nsert an image into a document or message 


nstall a program or [app] 


Judge whether you can trust information you find on 
he internet 


Scores 
20 30 40 50 60 70 80 
| do not think | could do this 
| have never done this but | could work out how to do this 
HB! know how to do this 
Sum 
8 Bilt 72 00 
8 24 [els) 00 
8 24 68 00 
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Figure F.13: Item map for the scale reflecting students’ ICT self-efficacy regarding the use of 
specialist applications 


Scores 


How well can you do each of these tasks when using ICT? 20 30 40 50 60 70 80 


Create a database (e.g., using [Microsoft Access ®]) 


Create a computer program, macro, or [app] 
(e.g., in [Basic, Visual Basic]) 


Set up a local area network of computers or other 
ICT 


| do not think | could do this 
[have never done this but | could work out how to do this 
HB | know how to do this 


Sum 
Create a database (e.g., using [Microsoft Access ®]) 28 26 00 
Build or edit awebpage 24 yXe) 00 


Create a computer program, macro, or [app] = 

2 : : : 42 17 00 
(e.g., in [Basic, Visual Basic]) 

Set up a local area network of computers or other 35 


ICT ae 
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Figure F.14: Item map for the scale reflecting students’ perceptions of positive outcomes of ICT for 
society 


Scores 


How much do you agree or disagree with the following 20 30 
statements about ICT? 


Advances in ICT usually improve people’s living 
conditions 


ICT helps us to understand the world better 


40 50 60 70 80 
ICT is valuable to society 


Strongly disagree Disagree 
WB Agree I Strongly agree 

Sum 
Advances in ICT usually improve people’s living 3 2 31 100 
conditions 
ICT helps us to understand the world better 2 le 100 
ICT is valuable to society 3 13 100 
Advances in ICT bring many social benefits 3 4 100 
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Figure F.15: Item map for the scale reflecting students’ perceptions of negative outcomes of ICT for 
society 


Scores 


How much do you agree or disagree with the following 20 30 40 50 60 70 80 
statements about ICT? 


Using ICT makes people more isolated in society 


With more ICT there will be fewer jobs 


Using ICT may be dangerous for people’s health 


Strongly disagree Disagree 
BB Agree I Strongly agree 
Sum 
Using ICT makes people more isolated in society 7 28 24 100 
With more ICT there will be fewer jobs 12 oT 18 100 
People spend far too much time using ICT 4 16 38 100 
Using ICT may be dangerous for people's health 8 23 25 100 
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Figure F.16: Item map for the scale reflecting students’ expectations of future ICT use for work and 
study 


Scores 


How much do you agree or disagree with the following 20 30 40 50 60 70 80 
statements about ICT? 


would like to study subjects related to ICT after 
secondary school] 


hope to find a job that involves advanced ICT — let 


Learning how to use ICT applications will help me to 
do the work | am interested in 


Strongly disagree Disagree 
WB Agree I Strongly agree 


Sum 


| would like to study subjects related to ICT after 
[secondary school] 


| hope to find a job that involves advanced ICT 18 31 100 


20 31 100 


Learning how to use ICT applications will help me to 


do the work | am interested in i a 


100 
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Figure F.17: Item map for the scale reflecting teachers’ ICT self-efficacy 


wn 
a 
lo} 
= 
oO 
wn 


Ww 
je) 


60 70 8 


js) 


How well can you do these tasks using ICT? 20 30 40 


Find useful teaching resources on the internet 


Contribute to a discussion forum/user group on the 
internet (e.g., a wiki or blog) 


Produce presentations (e.g., [PowerPoint® or a 
similar program]), with simple animation functions 


Use the internet for online purchases and payments 


Prepare lessons that involve the use of ICT by 
udents 


A) 


Using a spreadsheet program (e.g., [Microsoft Excel 
®]) for keeping records or analyzing data 


Assess student learning 


Collaborate with others using shared resources such 
as [Google Docs®], [Padlet] 


Use a learning management system (e.g., [Moodle], 
[Blackboard], [Edmodo]) 


| do not think | could do this 
| have never done this but | could work out how to do this 
HB | know how to do this 


Sum 
Find useful teaching resources on the internet 2 3 95 00 
Contribute toa discussion forum/user group on the j 7 58 00 
nternet (e.g,. a wiki or blog) 
Produce presentations (e.g., [PowerPoint® or a similar 5 12 84 00 
program]), with simple animation functions 
Use the internet for online purchases and payments 3 al isIo) 00 
Prepare lessons that involve the use of ICT by 3 18 78 00 
students 
Using a spreadsheet program (e.g., [Microsoft Excel 7 35 59 00 
®]) for keeping records or analyzing data © 
Assess student learning 3 18 As) 00 
Collaborate with others using shared resources such 7 35 00 
as [Google Docs®], [Padlet] 
Use a learning management system (e.g., [Moodle], 7 35 00 
[Blackboard], [Edmodo]) 
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Figure F.18: Item map for the scale reflecting teachers’ perceptions of positive outcomes when 
using ICT in teaching and learning 
Scores 
To what extent do you agree or disagree with the 20 30 40 50 60 70 80 


following practices and principles in relation to the use of 
ICT in teaching and learning? 


elps students develop greater interest in learning 


elps students to work at a level appropriate to 
heir learning needs 


elps students develop problem-solving skills 


Enables students to collaborate more effectively 


Helps students develop skills in planning and self- 
regulation of their work 


mproves academic performance of students. 


Enables students to access better sources of 


information 
Strongly disagree Disagree 
HB Agree I Strongly agree 
Sum 
elps students develop greater interest in learning a. 8 00 
elps students to work at a level appropriate to their 1 12 


earning needs 


DS) joe) 
je} 
je) 


K 
ie) 
io} 


elps students develop problem-solving skills 2 19) 


js 
wo 
je} 
je) 


Enables students to collaborate more effectively 1 DAL 


Helps students develop skills in planning and self- 


: : 2 23) 12 00 
regulation of their work - Xm 

mproves academic performance of students 2 2 00 
Enables students to access better sources of 1 7 29 00 


information 
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Figure F.19: Item map for the scale reflecting teachers’ perceptions of negative outcomes when 
using ICT in teaching and learning 


Scores 


To what extent do you agree or disagree with the 
following practices and principles in relation to the use of 
ICT in teaching and learning? 


20 30 40 50 60 70 


ioe) 
oO 


mpedes concept formation by students 


Results in students copying material from internet 
sources 


Distracts students from learning 


Results in poorer written expression among students 


Results in poorer calculation and estimation skills 
among students 


Limits the amount of personal communication 
among students 


Strongly disagree Disagree 
WB Agree If Strongly agree 
Sum 
mpedes concept formation by students 19 58 00 


Results in students copying material from internet 2 oy, 00 
sources 

Distracts students from learning 8 56 00 
Results in poorer written expression among students 6 42 


Results in poorer calculation and estimation skills 
among students 6 53 


jo) 
oO 


Limits the amount of personal communication among 
students 


jo) 
[o) 


BR R “ — oO 
| | 7 
je} 
is) 
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Figure F.20: Item map for the scale reflecting teachers’ perceptions of the availability of ICT 
resources at school 

Scores 
To what extent do you agree or disagree with the 20 30 40 50 60 70 80 
following statements about using ICT in teaching at your 
school? 


y school has sufficient ICT equipment 
.g., computers) 


io 


The computer equipment in our school is up-to-date 


y school has access to sufficient digital learning 
resources (e.g., learning software or [apps]) 


y school has good connectivity (e.g., fast speed) to 
the internet 


There is enough time to prepare lessons that 
incorporate ICT 


There is sufficient opportunity for me to develop 
expertise in ICT 


There is sufficient technical support to maintain ICT 


There is sufficient technical support to maintain ICT 1 34. 
resources 


resources 
Strongly disagree Disagree 
i Agree I Strongly agree 
Sum 
y school has sufficient ICT equipment 9 29 00 
(e.g., computers) 
The computer equipment in our school is up-to-date 9 30 00 
y school has access to sufficient digital learning ie 
ein g oe 5) 00 
resources (e.g., learning software or [apps]) 
y school has good connectivity (e.g., fast speed) to 2 30 16 00 
the internet 
[here is enough time to prepare lessons that 5 44 8 00 
incorporate ICT 
There is sufficient opportunity for me to develop 0 38 00 
expertise in ICT 
= 
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Figure F.21: Item map for the scale reflecting teachers’ perceptions of collaboration between 


teachers when using ICT 


To what extent do you agree or disagree with the 
following statements about your use of ICT in teaching and 
earning at your school? 


work together with other teachers on improving 
he use of ICT in classroom teaching 


collaborate with colleagues to develop ICT-based 
essons 


observe how other teachers use ICT in teaching 


discuss with other teachers how to use ICT in 
eaching topics 


share IC T-based resources with other teachers in 
my school 


work together with other teachers on improving the 
use of ICT in classroom teaching 


collaborate with colleagues to develop ICT-based 
essons 


observe how other teachers use ICT in teaching 


discuss with other teachers how to use ICT in 
eaching topics 


share ICT-based resources with other teachers in 
my school 
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Figure F.22: Item map for the scale reflecting teachers’ emphasis on developing ICT capabilities in 


class 


In your teaching the reference class in this school year, 
how much emphasis have you given to developing the 
following ICT-based capabilities in your students? 


To access information efficiently 


To display information for a given audience/purpose 


To evaluate the credibility of digital information 


To share digital information with others 


To use computer software to construct digital work 


products (e.g., presentations, documents, images, 
and diagrams) 


To provide digital feedback on the work of others 
(such as classmates) 


To explore a range of digital resources when 
searching for information 


To provide references for digital information sources 


To understand the consequences of making 
information publicly available online 


To access information efficiently 


To display information for a given audience/purpose 


To evaluate the credibility of digital information 


To share digital information with others 


To use computer software to construct digital work 
products (e.g., presentations, documents, images, 
and diagrams) 


To provide digital feedback on the work of others 
(such as classmates) 


To explore a range of digital resources when 
searching for information 


To provide references for digital information sources 


To understand the consequences of making 
information publicly available online 


29 
Scores 
20 30 40 50 60 70 80 
No emphasis Little emphasis 


HB Some emphasis 


I Strong emphasis 


Sum 
4 12 Oy 00 
6 16 Sil 00 
7 19) oo 00 
7 18 oD) 00 
12 22 ce) 00 
11 22 26 00 
12 22 30 00 
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Figure F.23: Item map for the scale reflecting teachers’ emphasis of teaching CT-related tasks in 
class 


Scores 
In your teaching of the reference class this school year, 
how much emphasis have you given to teaching the 20 30 40 50 60 70 
following skills? 


oo 
js) 


To display information in different ways 


To break a complex process into smaller parts 


To understand diagrams that describe or show real- 
world problems 


To plan tasks by setting out the steps needed to 
complete them 


To use tools making diagrams that help solve 
problems 


To use simulations to help understand or solve real- 
world problems 


To make flow diagrams to show the different parts 
of a process 


To record and evaluate data to understand and solve 
a problem 


To use real-world data to review and revise solutions 


to problems 
No emphasis Little emphasis 
HB Some emphasis I Strong emphasis 
Sum 

To display information in different ways 4 12 34 00 
To break a complex process into smaller parts 7 16 29 00 
To understand diagrams that describe or show real- 14 1 22 00 
world problems 
To plan tasks by setting out the steps needed to 7 V7 30 00 
complete them 
To use tools making diagrams that help solve 35 29 00 
problems 

o use simulations to help understand or solve real- 15 24 99 00 
world problems 
To make flow diagrams to show the different parts of 35 99 9 00 
a process 

o record and evaluate data to understand and solve 16 22 20 00 
a problem 

o use real-world data to review and revise solutions 15 al 09 00 
to problems 
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Jasmin Schiffer, graphic designer 
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Staff at IEA Hamburg 
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Sebastian Meyer, ICILS international data manager 

ichael Jung, ICILS international data manager (former) 
Ekaterina Mikheeva, ICILS deputy international data manager 
Lars Borchert, ICILS deputy international data manager (former) 
Sabine Meinck, head of research and analysis, and sampling units 
Sabine Tieck, research analyst (sampling, 
Sabine Weber, research analyst (sampling) 
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Adeoye Oyekan, research analyst 
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the test and questionnaire items, the assessment delivery system, and the web-based translation, 
scoring, and data-management modules. 


Staff at SoNET systems 

ike Janic, managing director 
Stephen Birchall, deputy CEO 

Erhan Halil, product development manager 
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Stephen Ainley, quality assurance 
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